Method and apparatus for data synchronization, data synchronization system

Abstract

The application discloses a data synchronization method and device, a data synchronization system. The method comprises the following steps: reading a database log by using a database pipeline process, wherein the log is used for recording the dependency relationship of data in the database, and is used for recording the query statement information of the update in the database, and the dependency relationship is used for indicating the flow direction of the data in the database; analyzing the log to obtain the flow direction of the data in the database, and generating a directed acyclic graph according to the flow direction of the data; storing the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizing the data in the database and the message middleware. The application solves the technical problem of long delay time of data synchronization caused by real-time access to the database for data synchronization.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a method and apparatus for data synchronization, and a data synchronization system. Background Technology

[0002] Traditional real-time access requires synchronizing data by connecting to a database and querying it, which puts enormous pressure on the database and is not very accurate. For example, if a log table stores thousands to tens of thousands of data entries per second, synchronizing data by directly querying the database will result in significant delays in database insertion and query operations, and may even cause the database to crash, greatly affecting data entry and failing to meet reporting requirements.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data synchronization method, apparatus, and system to at least solve the technical problem of long data synchronization delay caused by real-time access to a database.

[0005] According to one aspect of the embodiments of this application, a data synchronization method is provided, comprising: using a database pipeline process to read the log of a database, wherein the log is used to record the dependencies of data in the database and to record the query statement information updated in the database, and the dependencies are used to indicate the flow of data in the database; parsing the log to obtain the flow of data in the database, and generating a directed acyclic graph based on the flow of data; storing the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizing the data in the database with the data in the message middleware.

[0006] Optionally, before reading the database log, the data synchronization method includes: setting the database log mode to row mode; in row mode, listening to the database; and when data in the database is detected to have been modified, recording the data before modification, the data after modification, and the data modification process in the log.

[0007] Optionally, the database pipeline process is used to read the database logs, including: the database pipeline process sending a request message to the database through a preset protocol, wherein the request message is used to request to read the database logs; receiving the log data sent by the database, and converting the format of the log data into a target format.

[0008] Optionally, generating a directed acyclic graph (DAG) based on the data flow direction includes: determining multiple data processing units, using each data processing unit as a node of the DAG; using the data flow direction as an edge of the DAG; and generating the DAG based on the nodes and edges of the DAG.

[0009] Optionally, before reading the database logs, the data synchronization method includes: configuring the database address, determining the target data table and the start timestamp, wherein the target data table is the data table that needs to be synchronized between the database and the message middleware, and the start timestamp is used to indicate the time when the database and the message middleware begin synchronization.

[0010] Optionally, after synchronizing the data between the database and the message middleware, the data synchronization method includes: obtaining a request message from the terminal, wherein the request message is used to request data from the database; and sending the data corresponding to the request message to the terminal through the message middleware.

[0011] Optionally, the logs can be parsed, including parsing multiple data points in the logs simultaneously.

[0012] According to another aspect of the embodiments of this application, a data synchronization system is also provided, comprising: a relational database module for generating logs, wherein the logs are used to record the dependencies of data in the database and to record updated query statement information in the database, and the dependencies are used to indicate the flow of data in the database; a synchronization tool module for reading the logs and sending the logs to a distributed computing engine; a distributed computing engine module for parsing the logs to obtain the flow of data in the database and generating a directed acyclic graph based on the flow of data; storing the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizing the data in the database with the data in the message middleware.

[0013] According to another aspect of the embodiments of this application, a data synchronization apparatus is also provided, comprising: a reading module for reading a log of a database, wherein the log is used to record the dependencies of data in the database and to record updated query statement information in the database, and the dependencies are used to indicate the flow of data in the database; a parsing module for parsing the log to obtain the flow of data in the database and generating a directed acyclic graph based on the flow of data; and a synchronization module for storing the data in the database in a message middleware indicated by the directed acyclic graph and synchronizing the data in the database with the data in the message middleware.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to perform the above-described data synchronization method when it runs.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described data synchronization method during runtime.

[0016] In this embodiment, a database pipeline process is used to read the database logs. The logs record data dependencies and updated query information, with dependencies indicating the data flow direction. The logs are parsed to determine the data flow direction, and a directed acyclic graph (DAG) is generated based on this flow. The data is stored in a message middleware indicated by the DAG, and the database and message middleware are synchronized. By reading data from the database logs, real-time data exchange between the database and message middleware is achieved. This avoids direct terminal access to the database, reducing the pressure on the database. Furthermore, by parsing the logs in a distributed computing engine to form a DAG and performing stream processing on the data, the latency of data synchronization is reduced. This achieves the technical effect of real-time data synchronization between the database and the terminal using a message middleware, thus solving the problem of long latency caused by real-time database access. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data synchronization method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data synchronization method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a data synchronization system according to an embodiment of this application;

[0021] Figure 4 This is a structural diagram of a data synchronization device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0025] Pipeline process: A tool for synchronizing incremental data. It uses pipeline processes to synchronously store incremental data from the database to the target address.

[0026] Message middleware: A software system that provides message transmission for application systems in a network environment based on message passing technology.

[0027] Directed Acyclic Graph (DAG): A directed graph is a graph in which it is impossible to return to a vertex by traversing several edges from a vertex. In this embodiment of the application, it refers to a graph that starts from a data processing unit that processes the source data in the pipeline process and ends at a specified message middleware.

[0028] In related technologies, the terminal directly reads data from the database and synchronizes the data with the database. This results in high data synchronization latency and database crashes due to high database pressure. To address this issue, this embodiment uses a pipeline process to read database logs, captures changes in the database data in real time, synchronizes Data Manipulation Language (DML) operations from the database to a message middleware, and then synchronizes the data to the terminal via the message middleware. This solves the aforementioned problems, as detailed below.

[0029] According to an embodiment of this application, a method embodiment for data synchronization is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data synchronization method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0031] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data synchronization method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned application data synchronization method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0034] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0035] Figure 2 This is a flowchart of a data synchronization method provided according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0036] Step S202: The database pipeline process reads the database log, which is used to record the dependencies of data in the database and to record the query information of updates in the database. The dependencies are used to indicate the flow of data in the database.

[0037] In step S202, the logical log (binlog) of the relational database (MySQL) is read in real time through a pipeline process to obtain the data recorded in the binlog. The binlog records information related to data change operations in the database, such as the flow of data in the database, the operations performed when data changes occur, and newly added query statements (SQL statements) in the database.

[0038] Step S204: Parse the logs to obtain the data flow direction in the database, and generate a directed acyclic graph based on the data flow direction.

[0039] In step S204, the logs read in step S202 are parsed using a big data distributed computing engine to obtain the data flow direction, and a directed acyclic graph is generated based on the data in the database and the data flow direction.

[0040] Step S206: Store the data in the database in the message middleware indicated by the directed acyclic graph, and synchronize the data in the database with the data in the message middleware.

[0041] In step S206, the data read from the log, especially the incremental data of the database, is stored in the message middleware indicated by the directed acyclic graph, thereby realizing data synchronization between the database and the message middleware.

[0042] By following the steps above, we can reduce the pressure on the source database by reading the database's binlog file, and synchronize the data in the database, especially incremental data, to the message middleware with a latency of milliseconds for the terminal to read, thus reducing the data synchronization latency.

[0043] According to an optional embodiment of this application, the data synchronization method before reading the database log includes the following steps: setting the database log mode to row mode; in row mode, monitoring the database; when data in the database is detected to have been modified, recording the data before modification, the data after modification, and the data modification process in the log.

[0044] In this embodiment, before using the pipeline process to read the database's binlog, the database's log mode is set to row mode. In row mode, the binlog records the modified form of each row of data in the database tables, including its original form, modified form, and the specific operation performed to modify the data. When the pipeline process detects that data in the database has been modified, it modifies the corresponding data in the pipeline process based on the modified data, its original form, modified form, and the specific operation recorded in the binlog, thus synchronizing the pipeline process with the database. By setting the database's log mode to row mode, the binlog only records the modified statements without recording the context statements, reducing the amount of data transmitted during data synchronization and lowering data synchronization latency.

[0045] According to another optional embodiment of this application, reading database logs using a database pipeline process includes: the database pipeline process sending a request message to the database through a preset protocol, wherein the request message is used to request reading the database logs; receiving log data sent by the database, and converting the format of the log data into a target format.

[0046] In this embodiment, the pipeline process reads the binlog of the database using the following method: the MySQL database being read is the master database, and the pipeline process simulates a slave database of the master database (MySQL). Through the interaction protocol of the master database (MySQL) (i.e., the preset protocol), it sends a dump request to the master database (MySQL). Upon receiving the dump request, the master database (MySQL) sends incremental data from the binlog, i.e., the data that has changed, to the pipeline process. After receiving the incremental data from the binlog, the pipeline process encapsulates the data belonging to the same event according to the event corresponding to the data and converts it into the data format (i.e., the target format) in the pipeline process.

[0047] According to another optional embodiment of this application, generating a directed acyclic graph based on the data flow direction includes the following steps: determining multiple data processing units, and using each data processing unit as a node of the directed acyclic graph; using the data flow direction as an edge of the directed acyclic graph; and generating the directed acyclic graph based on the nodes and edges of the directed acyclic graph.

[0048] In this embodiment, the big data distributed computing engine determines the number of events corresponding to the data based on the encapsulated data in the pipeline process, and determines multiple data processing units corresponding to multiple time periods. Each data processing unit is used to process data belonging to one event. In the big data distributed computing engine, the data processing unit includes data source units, data processing units, and data sink units. These units are used as nodes in a directed acyclic graph (DAG), and the data flow direction, i.e., the data processing order, is used as the edges of the DAG to generate a DAG, representing the events corresponding to the data in the form of a DAG. Each DAG starts with the node corresponding to the data source unit and ends with the node corresponding to the data sink unit.

[0049] According to some preferred embodiments of this application, before reading the database logs, the data synchronization method includes: configuring the database address, determining the target data table and the start timestamp, wherein the target data table is the data table that needs to be synchronized between the database and the message middleware, and the start timestamp is used to indicate the time when the database and the message middleware begin synchronization.

[0050] In some preferred embodiments, before reading the binlog logs of the database through the pipeline process, the pipeline process is configured as follows: the address of the target database to be read, the name of the target data table to be synchronized between the message middleware and the database, and the start time of synchronization between the message middleware and the database are configured. After configuration, the pipeline process will perform data synchronization between the databases according to the above configuration when data changes occur in the target data table. In addition, when configuring the pipeline process, the protocol for data transmission between the pipeline process and the database is also configured, enabling data transmission between the pipeline process and the database. The method provided in this embodiment allows the pipeline process to read data tables in different databases without needing to reconfigure for a single database or a single data table.

[0051] According to an optional embodiment of this application, after synchronizing the data between the database and the message middleware, the data synchronization method includes: obtaining a request message from the terminal, wherein the request message is used to request data from the database; and sending data corresponding to the request message to the terminal through the message middleware.

[0052] In this embodiment, after synchronizing the database and message middleware using the above method, the terminal no longer reads data from the database. When the database receives a data request message from the terminal, it directly sends the data to the terminal from the message middleware in the caching layer, which is equivalent to the terminal directly reading the data from the message middleware. Furthermore, if the terminal does not temporarily request data from the database, the message middleware still synchronizes data with the database in real time, temporarily caching the data in the message middleware. When the terminal re-reads the data, it can immediately retrieve the data from the message middleware, reducing the data synchronization latency.

[0053] According to some other preferred embodiments of this application, parsing the log includes: parsing multiple data in the log simultaneously.

[0054] In some other preferred embodiments, a parallel approach is used to parse multiple data points in the log simultaneously, rather than parsing each data point one by one, which improves the speed of data parsing and helps reduce the latency of data synchronization.

[0055] Figure 3This is a schematic diagram of a data synchronization system provided in an embodiment of this application, including: a relational database module 30, used to generate logs, wherein the logs are used to record the dependencies of data in the database and to record the query statement information updated in the database, and the dependencies are used to indicate the flow of data in the database; a synchronization tool module 32, used to read the logs and send the logs to a distributed computing engine; a distributed computing engine module 34, used to parse the logs to obtain the flow of data in the database and generate a directed acyclic graph based on the flow of data; storing the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizing the data in the database with the data in the message middleware.

[0056] like Figure 3 As shown, the relational database module 30 is communicatively connected to the synchronization tool module 32. The relational database module 30 receives a dump request message from the synchronization tool module 32 requesting log data, and transmits the generated binlog log data to the synchronization tool module 32 via a transmission protocol. The synchronization tool module 32 is communicatively connected to the distributed computing engine module 34 and receives the log data encapsulated and processed by the synchronization tool module 32. The distributed computing engine module 34 parses the received log data, generates a directed acyclic graph, and stores the received log data in the middleware indicated by the directed acyclic graph, thereby achieving data synchronization between the database and the message middleware.

[0057] It should be noted that the modules in the above data synchronization system can be program modules (e.g., a set of program instructions to implement a specific function) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.

[0058] Figure 4 This is a structural diagram of a data synchronization device according to an embodiment of this application, including: a reading module 40, used to read the logs of the database, wherein the logs are used to record the dependencies of data in the database and to record the query statement information updated in the database, and the dependencies are used to indicate the flow of data in the database; a parsing module 42, used to parse the logs to obtain the flow of data in the database, and to generate a directed acyclic graph based on the flow of data; and a synchronization module 44, used to store the data in the database in a message middleware indicated by the directed acyclic graph, and to synchronize the data in the database with the data in the message middleware.

[0059] When the aforementioned data synchronization device is working, the reading module 40 reads incremental data from the logs in the database, i.e., the data that has changed. It reads the data's form before the change, its form after the change, the specific operations performed when the data changes, the query statements used to perform the data change operations, and the data flow direction. The parsing module 42 parses the read log data to obtain the data flow direction recorded in the log data and generates a directed acyclic graph (DAG) based on this flow. This DAG transmits the data operation logic rather than the data itself, greatly reducing server resource waste, lowering data synchronization latency, and increasing data visibility by clearly showing the data flow direction and the operations performed on the data at each node of the DAG. The synchronization module 44 stores the data read from the database in the message middleware indicated by the aforementioned DAG, achieving data synchronization between the database and the message middleware.

[0060] It should be noted that, Figure 4 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0061] This application also provides a non-volatile storage medium storing a program, wherein the program controls the device where the non-volatile storage medium is located to perform the above-mentioned data synchronization method when it runs.

[0062] The aforementioned non-volatile storage medium is used to store programs that perform the following functions: reading database logs using a database pipeline process, wherein the logs are used to record the dependencies of data in the database and to record the query information updated in the database, and the dependencies are used to indicate the flow of data in the database; parsing the logs to obtain the flow of data in the database, and generating a directed acyclic graph based on the flow of data; storing the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizing the data in the database with the data in the message middleware.

[0063] This application also provides an electronic device, including a memory and a processor, the processor being used to run a program stored in the memory, wherein the program executes the above-described data synchronization method during runtime.

[0064] The processor in the aforementioned electronic device is used to run a program that performs the following functions: reads the database log using a database pipeline process, wherein the log is used to record the dependencies of data in the database and to record the query information updated in the database, and the dependencies are used to indicate the flow of data in the database; parses the log to obtain the flow of data in the database, and generates a directed acyclic graph based on the flow of data; stores the data in the database in a message middleware indicated by the directed acyclic graph, and synchronizes the data in the database with the data in the message middleware.

[0065] It should be noted that the modules in the above-mentioned data synchronization device can be program modules (e.g., a set of program instructions to implement a certain function) or hardware modules. For the latter, they can be manifested in the following forms, but are not limited to these: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0066] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0067] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0068] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0071] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0072] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of data synchronization, characterized by, include: A database pipeline process is used to read the database logs, wherein the logs are used to record the dependencies of data in the database and to record the updated query information in the database, and the dependencies are used to indicate the flow of data in the database; Parsing the logs to obtain the data flow direction in the database, and generating a directed acyclic graph (DAG) based on the data flow direction, includes: determining multiple data processing units, using each of the multiple data processing units as a node of the DAG; using the data flow direction as an edge of the DAG; generating the DAG based on the nodes and edges of the DAG; the data processing units are used to process the source data in the database pipeline process; The data in the database is stored in the message middleware indicated by the directed acyclic graph, and the data in the database is synchronized with the data in the message middleware.

2. The method of claim 1, wherein, Before reading the database logs, the method includes: Set the log mode of the database to row mode; In the row mode, the database is monitored; when data in the database is detected to have been modified, the data before modification, the data after modification, and the modification process are recorded in the log.

3. The method of claim 1, wherein, The database pipeline process reads the database logs, including: The database pipeline process sends a request message to the database through a preset protocol, wherein the request message is used to request to read the database logs; Receive log data sent by the database and convert the format of the log data into the target format.

4. The method of claim 1, wherein, Before reading the database logs, the method includes: Configure the address of the database, determine the target data table and the start timestamp, wherein the target data table is the data table that needs to be synchronized between the database and the message middleware, and the start timestamp is used to indicate the time when the database and the message middleware start synchronizing.

5. The method of claim 1, wherein, After synchronizing the database with the data of the message middleware, the method includes: Obtain a request message from the terminal, wherein the request message is used to request data from the database; The message middleware sends data corresponding to the request message to the terminal.

6. The method of claim 1, wherein, Parsing the log includes parsing multiple data points in the log simultaneously.

7. A data synchronization system, characterized in that, include: The module includes a relational database module, a synchronization tool module, a distributed computing engine module, and a message middleware module. The relational database module is used to generate logs, wherein the logs are used to record the dependencies of data in the database and to record the updated query information in the database, and the dependencies are used to indicate the flow of data in the database; The synchronization tool module is used to read the logs and send the logs to the distributed computing engine; The distributed computing engine module is used to parse the logs to obtain the flow direction of data in the database, and generate a directed acyclic graph based on the flow direction of the data; store the data in the database in the message middleware indicated by the directed acyclic graph, and synchronize the data in the database with the data in the message middleware; The distributed computing engine module is further configured to determine multiple data processing units, use each of the multiple data processing units as a node of the directed acyclic graph (DAG), use the flow direction of the data as an edge of the DAG, generate the DAG based on the nodes and edges of the DAG, and the data processing unit is configured to process the source data in the database pipeline process.

8. A data synchronization device, characterized in that, include: The reading module is used to read the database logs, wherein the logs are used to record the dependencies of data in the database and to record the updated query statement information in the database, and the dependencies are used to indicate the flow of data in the database; A parsing module is used to parse the logs to obtain the flow direction of data in the database, and generate a directed acyclic graph (DAG) based on the data flow direction. This includes: determining multiple data processing units, using each of the multiple data processing units as a node of the DAG; using the data flow direction as an edge of the DAG; and generating the DAG based on the nodes and edges of the DAG. The data processing units are used to process the source data in the database pipeline process. The synchronization module is used to store the data in the database in the message middleware indicated by the directed acyclic graph, and to synchronize the data in the database with the data in the message middleware.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the data synchronization method according to any one of claims 1 to 6.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data synchronization method according to any one of claims 1 to 6.

Images