Data synchronization method, system, device and storage medium supporting breakpoint resumption

Abstract

The present invention relates to a data synchronization method, system, device and storage medium supporting breakpoint resumption, and the technical scheme thereof mainly includes: obtaining log file data of a source database; building a table according to the log file data; writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table; transmitting the data written with the data nodes to a target database, and in the process of transmitting the data written with the data nodes to the target database, in the case of transmitting the data nodes, judging whether information of receiving the data nodes fed back by the target database is received, and in the case of receiving the information fed back by the target database, storing the location information of the data nodes and the corresponding storage time; the present application has the effect of quickly retransmitting the data that has not been successfully transmitted after the breakpoint, avoiding resynchronization of the previously transmitted data, and greatly improving the efficiency of data synchronization.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and more specifically, to a data synchronization method, system, device and storage medium supporting breakpoint resume transmission. Background Art

[0002] Existing big data platforms usually use offline data synchronization, but there are scenarios in each business system that require historical data recalculation and the amount of historical data is huge. If the historical data is still synchronized offline, it will take up a lot of computing resources and it is difficult to define the scope of data synchronization, resulting in a certain error between the historical data stored in the data warehouse and the actual business data. The data accuracy cannot support business needs in certain scenarios. In addition, due to other high real-time requirements such as business or large screens, the platform has put forward data real-time requirements. Summary of the invention

[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a data synchronization method, system, device and storage medium that supports breakpoint resumption, which has the functional advantage of being able to quickly retransmit data that has not been successfully transmitted after a breakpoint, thereby avoiding the need to resynchronize previously transmitted data.

[0004] The above technical objectives of the present invention are achieved through the following technical solutions:

[0005] A data synchronization method supporting breakpoint resumable transmission, comprising:

[0006] Get the log file data of the source database;

[0007] Create a table based on the log file data;

[0008] Writing corresponding data nodes into the data in the table according to data features of the data in the table;

[0009] The data written with the data node is transmitted to the target database. When the data node is transmitted during the process of transmitting the data written with the data node to the target database, it is determined whether information of receiving the data node is received from the target database. If the information is received from the target database, the location information of the data node and the corresponding storage time are stored.

[0010] receiving a breakpoint resume request, and determining a request time of the breakpoint resume request;

[0011] Determine, according to the request time, a target data node corresponding to a storage time closest to the request time point;

[0012] The data that has not been transmitted to the target database is determined according to the target data node and the data written to the data node, and is further transmitted to the target database.

[0013] Optionally, the data features include: data peaks and data troughs; and writing corresponding data nodes into the data in the table according to the data features of the data in the table includes:

[0014] Determine whether the data in the table has a data peak, and if there is a data peak, write a corresponding data node at the data peak;

[0015] It is determined whether the data in the table has a data trough, and if so, a corresponding data node is written at the data trough.

[0016] Optionally, the data feature further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node into the data in the table according to the data feature of the data in the table also includes:

[0017] Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold. If the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, calculate the data length based on the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and write the corresponding data node at the data writing point.

[0018] Optionally, the creating a table according to the log file data includes:

[0019] Parsing the log file data to obtain the database name, table name, operation type, primary key and all field values;

[0020] Create a table based on the database name, table name, operation type, primary key and all field values.

[0021] Optionally, after creating a table according to the log file data, the method further includes:

[0022] Dividing the data in the table into multiple data rows according to the primary key;

[0023] Each data row is divided into a plurality of data blocks according to the preset configuration.

[0024] Optionally, the transmitting the data written in the data node to the target database includes:

[0025] Reading all data blocks in each of the data rows in sequence according to a timestamp, wherein the timestamp is obtained from the log file data;

[0026] When the front end of the data block is read, a first binlog position is marked at the front end of the data block, wherein the first binlog position is position information of the front end of the data block obtained from the log file data;

[0027] In the case where the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is position information of the back end of the data block obtained from the log file data;

[0028] The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

[0029] Optionally, also include:

[0030] When data in the source database changes during the process of transferring data written with data nodes to the target database, the data row corresponding to the changed data is determined based on the primary key corresponding to the changed data, the position of the changed data is compared with all the first binlog positions and all the second binlog positions of the data row to determine the data block corresponding to the changed data, and the data of the data block corresponding to the changed data is changed based on the primary key corresponding to the changed data.

[0031] A data synchronization system supporting breakpoint-resume transmission, comprising:

[0032] A data acquisition module is used to acquire log file data from a source database;

[0033] A data table building module, used to build a table based on the log file data;

[0034] A node writing module, used for writing corresponding data nodes into the data in the table according to the data characteristics of the data in the table;

[0035] a transmission judgment module, used to transmit the data written with the data node to the target database, and in the case of transmitting the data node during the process of transmitting the data written with the data node to the target database, judge whether the information of receiving the data node fed back by the target database is received, and in the case of receiving the information fed back by the target database, store the location information of the data node and the corresponding storage time;

[0036] A request receiving module, used for receiving a breakpoint resume request and determining a request time of the breakpoint resume request;

[0037] A node determination module, used to determine, according to the request time, a target data node corresponding to a storage time closest to the request time point;

[0038] The data retransmission module is used to determine the data that has not been transmitted to the target database according to the target data node and the data written to the data node, and continue to transmit the data to the target database.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0041] In summary, the present invention has the following beneficial effects: after a breakpoint, data that has not been successfully transmitted can be quickly retransmitted, avoiding resynchronization of previously transmitted data, and greatly improving the efficiency of data synchronization; and compared to dividing data into fixed segments, by writing corresponding data nodes at data peaks and data troughs, and determining data consumption time through real-time network transmission speed, and then controlling the data transmission time between adjacent data nodes within a preset time threshold based on the data consumption time and the preset time threshold, it is possible to divide data according to the characteristics of the data, and dynamically determine how much data to process as a unit, which is more suitable for real-time synchronous transmission of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a data synchronization method supporting breakpoint resume provided by the present invention;

[0043] Figure 2 It is a structural block diagram of a data synchronization system supporting breakpoint resume transmission provided by the present invention;

[0044] Figure 3 It is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0046] In the present invention, unless otherwise expressly specified and limited, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0047] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0048] The present invention provides a data synchronization method supporting breakpoint resume transmission, such as Figure 1 As shown, including:

[0049] Step 100: Obtain log file data of the source database. Specifically, the source database adopts a relational database, and the log file data is obtained by directly monitoring the binlog log of the relational database (that is, the source database). Binlog is a binary file used to record SQL statement information updated by the user on the database.

[0050] Step 200: Create a table based on the log file data;

[0051] Step 300: Write corresponding data nodes into the data in the table according to the data features of the data in the table;

[0052] Step 400: Transmit the data written with the data node to the target database. In the process of transmitting the data written with the data node to the target database, in the case of transmitting the data node, determine whether the information of receiving the data node is received from the target database. If the information is received from the target database, it means that the data before the data node has been transmitted to the target database. Store the location information of the data node and the corresponding storage time. The storage time is the time point when the location information of the data node is stored.

[0053] Step 500: receiving a breakpoint resume request and determining a request time of the breakpoint resume request;

[0054] Step 600: Determine the target data node corresponding to the storage time closest to the request time according to the request time; in actual applications, before a breakpoint occurs, there may be location information corresponding to multiple data nodes stored, and the data node corresponding to the storage time closest to the request time is determined according to the request time of the breakpoint resume request, that is, the location information of the data node last stored before the breakpoint is determined and used as the target data node;

[0055] Step 700: determine the data that has not been transmitted to the target database based on the target data node and the data written to the data node, and continue to transmit it to the target database, wherein the data written to the data node specifically refers to the data generated by writing the corresponding data node in the data in the table according to the data characteristics of the data in the table.

[0056] In actual applications, steps 100 to 400 are steps of normal transmission performed when data transmission is normal, that is, when data transmission is not interrupted due to network interruption or power failure, etc., and when data transmission is interrupted (that is, breakpoint) and then the data transmission task is restarted, that is, after a breakpoint resume request is issued, steps 500 to 700 are executed. The data synchronization method supporting breakpoint resume transmission of the present application can quickly retransmit the data that has not been successfully transmitted after the breakpoint, avoiding resynchronization of the previously transmitted data, and greatly improving the efficiency of data synchronization.

[0057] Further, the data features include: data peaks and data troughs; and writing corresponding data nodes into the data in the table according to the data features of the data in the table includes:

[0058] Determine whether the data in the table has a data peak, and if so, write a corresponding data node at the data peak; specifically, if there are multiple data peaks, write a corresponding data node at each data peak;

[0059] It is determined whether the data in the table has a data trough. If there is a data trough, a corresponding data node is written at the data trough; specifically, if there are multiple data troughs, a data node corresponding to each data trough is written at the data trough.

[0060] In practical applications, compared with dividing data into fixed segments, writing corresponding data nodes at data peaks and data troughs can achieve data division based on data characteristics, which is more suitable for real-time synchronous transmission of data.

[0061] Furthermore, the data characteristics also include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the corresponding data node is written into the data in the table according to the data characteristics of the data in the table, and further includes:

[0062] Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold. The preset time threshold represents a time period. When the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, it means that the data consumption time between adjacent data nodes in the data nodes determined according to the data peaks and data troughs is too long. If data breakpoints appear between these adjacent data nodes, it is not conducive to efficient data synchronization. Therefore, the data length is calculated according to the preset time threshold and the real-time network transmission speed, the data length is added to the previous data node in the adjacent data nodes to obtain the data writing point, and the corresponding data node is written at the data writing point.

[0063] Specifically, the data consumption time is determined according to the real-time network transmission speed, and then the data transmission time between adjacent data nodes is controlled within the preset time threshold according to the data consumption time and the preset time threshold. It can dynamically decide how much data to process as a unit, so that after a data transmission breakpoint occurs, there is no need to spend too much time to retransmit the previously transmitted data from the target data node, further improving the efficiency of data synchronization.

[0064] Further, the creating a table according to the log file data includes:

[0065] Parsing the log file data to obtain the database name, table name, operation type, primary key and all field values;

[0066] Create a table based on the database name, table name, operation type, primary key and all field values.

[0067] Further, after the table is created according to the log file data, the method further includes:

[0068] Dividing the data in the table into multiple data rows according to the primary key;

[0069] Each data row is divided into a plurality of data blocks according to the preset configuration.

[0070] In actual applications, the data in the table is divided into multiple data rows according to the primary key. During the data synchronization process, the data in multiple data rows can be transmitted simultaneously to achieve multi-concurrent transmission of data, thereby further improving the effect of data synchronization transmission. The preset configuration can be a preset data length. Each data row is divided into multiple data blocks according to the preset configuration. When the data in the source database changes, the data blocks corresponding to the changed data can be directly corrected to achieve real-time update of the data.

[0071] Furthermore, the transmitting of the data written with the data node to the target database includes:

[0072] Reading all data blocks in each of the data rows in sequence according to a timestamp, wherein the timestamp is obtained from the log file data;

[0073] When the front end of the data block is read, a first binlog position is marked at the front end of the data block, wherein the first binlog position is position information of the front end of the data block obtained from the log file data;

[0074] In the case where the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is position information of the back end of the data block obtained from the log file data;

[0075] The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

[0076] Furthermore, it also includes:

[0077] In the process of transmitting the data written with the data node to the target database, if the source database has data changes, the data changes include: data addition, data deletion and data modification, determining the data row corresponding to the changed data according to the primary key corresponding to the changed data, comparing the position of the changed data with all the first binlog positions and all the second binlog positions of the data row to obtain a comparison result, determining the data block corresponding to the changed data according to the comparison result, and if the comparison result is that the position of the changed data is between a first binlog position and a second binlog position corresponding to the first binlog position, determining the data block corresponding to the changed data to be the first binlog position and the data block corresponding to the first binlog position, changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data, and if the changed data is added, deleted or modified, performing corresponding addition, deletion or modification operations on the data in the corresponding database.

[0078] By marking the data block, the first binlog position and the second binlog position, the speed of finding the location of the changed data in the table is accelerated, which not only enables real-time update of data but also improves the efficiency of data synchronization.

[0079] The data synchronization method supporting breakpoint resumption of the present invention can quickly retransmit data that has not been successfully transmitted after a breakpoint, avoiding resynchronization of previously transmitted data, and greatly improving the efficiency of data synchronization; and compared with the fixed segmentation of data, by writing corresponding data nodes at data peaks and data troughs, and determining data consumption time through real-time network transmission speed, and then controlling the data transmission time between adjacent data nodes within a preset time threshold based on the data consumption time and the preset time threshold, it is possible to divide the data according to the characteristics of the data, and can dynamically determine how much data to process as a unit, which is more suitable for real-time synchronous transmission of data.

[0080] like Figure 2 As shown, the present invention also provides a system for building and distributing an installation package of an application, comprising:

[0081] The data acquisition module 10 is used to acquire the log file data of the source database;

[0082] A data table building module 20, used to build a table according to the log file data;

[0083] A node writing module 30, used to write corresponding data nodes into the data in the table according to data features of the data in the table;

[0084] The transmission judgment module 40 is used to transmit the data written with the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written with the data node to the target database, judge whether the information of receiving the data node is received from the target database, and in the case of receiving the information from the target database, store the location information of the data node and the corresponding storage time;

[0085] A request receiving module 50 is used to receive a breakpoint resume request and determine a request time of the breakpoint resume request;

[0086] A node determination module 60, configured to determine, according to the request time, a target data node corresponding to a storage time closest to the request time point;

[0087] The data retransmission module 70 is used to determine the data that has not been transmitted to the target database according to the target data node and the data written to the data node, and continue to transmit the data to the target database.

[0088] For the specific definition of the construction and distribution system of the application installation package, please refer to the definition of the construction and distribution method of the application installation package above, which will not be repeated here. The various modules of the construction and distribution system of the above-mentioned application installation package can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0089] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for building and distributing an installation package of an application is implemented.

[0090] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0092] Get the log file data of the source database;

[0093] Create a table based on the log file data;

[0094] Writing corresponding data nodes into the data in the table according to data features of the data in the table;

[0095] The data written with the data node is transmitted to the target database. When the data node is transmitted during the process of transmitting the data written with the data node to the target database, it is determined whether information of receiving the data node is received from the target database. If the information is received from the target database, the location information of the data node and the corresponding storage time are stored.

[0096] receiving a breakpoint resume request, and determining a request time of the breakpoint resume request;

[0097] Determine, according to the request time, a target data node corresponding to a storage time closest to the request time point;

[0098] The data that has not been transmitted to the target database is determined according to the target data node and the data written to the data node, and is further transmitted to the target database.

[0099] In one embodiment, the data features include: data peaks and data troughs; and writing corresponding data nodes into the data in the table according to the data features of the data in the table includes:

[0100] Determine whether the data in the table has a data peak, and if there is a data peak, write a corresponding data node at the data peak;

[0101] It is determined whether the data in the table has a data trough, and if so, a corresponding data node is written at the data trough.

[0102] In one embodiment, the data feature further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the corresponding data node is written into the data in the table according to the data feature of the data in the table, and further includes:

[0103] Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold. If the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, calculate the data length based on the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and write the corresponding data node at the data writing point.

[0104] In one embodiment, the step of creating a table based on the log file data includes:

[0105] Parsing the log file data to obtain the database name, table name, operation type, primary key and all field values;

[0106] Create a table based on the database name, table name, operation type, primary key and all field values.

[0107] In one embodiment, after the table is created according to the log file data, the method further includes:

[0108] Dividing the data in the table into multiple data rows according to the primary key;

[0109] Each data row is divided into a plurality of data blocks according to the preset configuration.

[0110] In one embodiment, the transmitting the data written in the data node to the target database includes:

[0111] Reading all data blocks in each of the data rows in sequence according to a timestamp, wherein the timestamp is obtained from the log file data;

[0112] When the front end of the data block is read, a first binlog position is marked at the front end of the data block, wherein the first binlog position is position information of the front end of the data block obtained from the log file data;

[0113] In the case where the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is position information of the back end of the data block obtained from the log file data;

[0114] The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

[0115] In one embodiment, it further includes:

[0116] When data in the source database changes during the process of transferring data written with data nodes to the target database, the data row corresponding to the changed data is determined based on the primary key corresponding to the changed data, the position of the changed data is compared with all the first binlog positions and all the second binlog positions of the data row to determine the data block corresponding to the changed data, and the data of the data block corresponding to the changed data is changed based on the primary key corresponding to the changed data.

[0117] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0118] Get the log file data of the source database;

[0119] Create a table based on the log file data;

[0120] Writing corresponding data nodes into the data in the table according to data features of the data in the table;

[0121] The data written with the data node is transmitted to the target database. When the data node is transmitted during the process of transmitting the data written with the data node to the target database, it is determined whether information of receiving the data node is received from the target database. If the information is received from the target database, the location information of the data node and the corresponding storage time are stored.

[0122] receiving a breakpoint resume request, and determining a request time of the breakpoint resume request;

[0123] Determine, according to the request time, a target data node corresponding to a storage time closest to the request time point;

[0124] The data that has not been transmitted to the target database is determined according to the target data node and the data written to the data node, and is further transmitted to the target database.

[0125] In one embodiment, the data features include: data peaks and data troughs; and writing corresponding data nodes into the data in the table according to the data features of the data in the table includes:

[0126] Determine whether the data in the table has a data peak, and if there is a data peak, write a corresponding data node at the data peak;

[0127] It is determined whether the data in the table has a data trough, and if so, a corresponding data node is written at the data trough.

[0128] In one embodiment, the data feature further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the corresponding data node is written into the data in the table according to the data feature of the data in the table, and further includes:

[0129] Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold. If the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, calculate the data length based on the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and write the corresponding data node at the data writing point.

[0130] In one embodiment, the step of creating a table based on the log file data includes:

[0131] Parsing the log file data to obtain the database name, table name, operation type, primary key and all field values;

[0132] Create a table based on the database name, table name, operation type, primary key and all field values.

[0133] In one embodiment, after the table is created according to the log file data, the method further includes:

[0134] Dividing the data in the table into multiple data rows according to the primary key;

[0135] Each data row is divided into a plurality of data blocks according to the preset configuration.

[0136] In one embodiment, the transmitting the data written in the data node to the target database includes:

[0137] Reading all data blocks in each of the data rows in sequence according to a timestamp, wherein the timestamp is obtained from the log file data;

[0138] When the front end of the data block is read, a first binlog position is marked at the front end of the data block, wherein the first binlog position is position information of the front end of the data block obtained from the log file data;

[0139] In the case where the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is position information of the back end of the data block obtained from the log file data;

[0140] The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

[0141] In one embodiment, it further includes:

[0142] When data in the source database changes during the process of transferring data written with data nodes to the target database, the data row corresponding to the changed data is determined based on the primary key corresponding to the changed data, the position of the changed data is compared with all the first binlog positions and all the second binlog positions of the data row to determine the data block corresponding to the changed data, and the data of the data block corresponding to the changed data is changed based on the primary key corresponding to the changed data.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A data synchronization method supporting breakpoint resuming, characterized in that: include: Get the log file data of the source database; Create a table based on the log file data; Write corresponding data nodes into the data in the table according to data features of the data in the table; the data features include data peaks, data troughs and data consumption time; the data nodes are logical marking points for identifying data transmission breakpoints; The data block with the data node written in it is transmitted to the target database. During the transmission process, if the target database receives feedback that the data block contains the information of the data node, the location information of the data node and the corresponding storage time are stored; receiving a breakpoint resume request, and determining a request time of the breakpoint resume request; Determine, according to the request time, a target data node corresponding to a storage time closest to the request time point; The data that has not been transmitted to the target database is determined according to the target data node and the data block written to the data node, and is further transmitted to the target database.

2. The data synchronization method supporting breakpoint resuming according to claim 1, characterized in that: Writing corresponding data nodes into the data in the table according to the data features of the data in the table includes: Determine whether the data in the table has a data peak, and if there is a data peak, write a corresponding data node at the data peak; It is determined whether the data in the table has a data trough, and if so, a corresponding data node is written at the data trough.

3. The data synchronization method supporting breakpoint resume according to claim 2, characterized in that: The data consumption time is determined according to the real-time network transmission speed; the corresponding data node is written into the data in the table according to the data characteristics of the data in the table, and further includes: Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold. If the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, calculate the data length based on the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and write the corresponding data node at the data writing point.

4. The data synchronization method supporting breakpoint resume according to claim 1, characterized in that: The step of building a table according to the log file data includes: Parsing the log file data to obtain the database name, table name, operation type, primary key and all field values; Create a table based on the database name, table name, operation type, primary key and all field values.

5. The data synchronization method supporting breakpoint resuming according to claim 4, characterized in that: After the table is created according to the log file data, the method further includes: Dividing the data in the table into multiple data rows according to the primary key; Each data row is divided into a plurality of data blocks according to a preset configuration, and the preset configuration is a preset data length.

6. The data synchronization method supporting breakpoint resuming according to claim 4, characterized in that: The step of transmitting the data block written with the data node to the target database includes: Reading all data blocks in each of the data rows in sequence according to a timestamp, wherein the timestamp is obtained from the log file data; When the front end of the data block is read, a first binlog position is marked at the front end of the data block, wherein the first binlog position is position information of the front end of the data block obtained from the log file data; In the case where the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is position information of the back end of the data block obtained from the log file data; The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

7. The data synchronization method supporting breakpoint resuming according to claim 6, characterized in that: Also includes: When data in the source database changes during the process of transferring data written with data nodes to the target database, the data row corresponding to the changed data is determined based on the primary key corresponding to the changed data, the position of the changed data is compared with all the first binlog positions and all the second binlog positions of the data row to determine the data block corresponding to the changed data, and the data of the data block corresponding to the changed data is changed based on the primary key corresponding to the changed data.

8. A data synchronization system supporting breakpoint resuming, characterized in that: include: A data acquisition module is used to acquire log file data from a source database; A data table building module, used to build a table based on the log file data; A node writing module, used for writing corresponding data nodes into the data in the table according to data characteristics of the data in the table; the data characteristics include data peaks, data troughs and data consumption time; the data nodes are logical marking points for identifying data transmission breakpoints; A transmission judgment module is used to transmit the data block written with the data node to the target database. During the transmission process, if the target database receives feedback that the data block contains the information of the data node, the location information of the data node and the corresponding storage time are stored; A request receiving module, used for receiving a breakpoint resume request and determining a request time of the breakpoint resume request; A node determination module, used to determine, according to the request time, a target data node corresponding to a storage time closest to the request time point; The data retransmission module is used to determine the data that has not been transmitted to the target database according to the target data node and the data written to the data node, and continue to transmit the data to the target database.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images