Structured data synchronization method and system based on paging and data persistence

Abstract

The invention discloses a structured data synchronization method and system based on paging and data persistence, and relates to the technical field of data synchronis.The method comprises the steps that a paging method based on time fields and a paging method based on database paging are constructed; acquiring to-be-synchronized data, selecting a paging method, setting paging parameters to execute corresponding query statements, and acquiring to-be-synchronized paging data; caching the to-be-synchronized paging data, executing a corresponding query statement on the to-be-synchronized paging data, and caching the to-be-synchronized paging data; and after the system is restarted or fails, on the basis of the cached query result, continuing to execute the to-be-synchronized paging data which is not completed last time, so as to realize breakpoint resume. According to the method, limited resources can be used for data synchronization, and after the data synchronization process is abnormal, rapid recovery can be realized, and breakpoint resume is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a structured data synchronization method and system based on paging and data persistence. Background Art

[0002] In the era of big data, data comes from a wide range of sources, including social media, IoT devices, enterprise applications, etc. In order to fully utilize the value of data, it is necessary to aggregate the data together for analysis. During the data aggregation process, if the amount of data is too large, one-time synchronization of data may encounter abnormal situations, such as network failure, insufficient disk space, etc. The traditional solution is to clear the data and run again, but this method wastes a lot of resources and time.

[0003] Therefore, how to quickly handle exceptions and implement breakpoint resumption when data synchronization exceptions occur becomes a key technology. Summary of the invention

[0004] In view of the needs and shortcomings of current technological development, the present invention provides a structured data synchronization method and system based on paging and data persistence, which is suitable for data synchronization under relatively limited resource conditions. After an exception occurs in the data synchronization task, it can quickly recover and resume the data from the breakpoint, thereby improving the performance and stability of data synchronization.

[0005] In the first aspect, the present invention provides a structured data synchronization method based on paging and data persistence. The technical solution adopted to solve the above technical problems is as follows:

[0006] A structured data synchronization method based on paging and data persistence includes the following steps:

[0007] S1. Construct a paging method based on time fields and a paging method based on database paging;

[0008] S2. Obtain the data to be synchronized, select the paging method, set the paging parameters, execute the corresponding query statement, and obtain the paging data to be synchronized;

[0009] S3: cache the paging data to be synchronized, execute the corresponding query statement on the paging data to be synchronized and cache it;

[0010] S4. After the system is restarted or fails, the paging data to be synchronized that was not completed last time is continued to be executed based on the cached query results to achieve breakpoint resume.

[0011] Optionally, the step S1 involved specifically includes:

[0012] When the data to be synchronized contains an incremental timestamp or date field, a paging method based on the time field is constructed. The paging method based on the time field includes four paging parameters: time range, data volume per page, time period segmentation, and query statement.

[0013] When there is no incremental field in the data to be synchronized, a paging method based on database paging is constructed. The paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset, and query statement.

[0014] Optionally, the step S3 involved specifically includes:

[0015] Persistently store the paging data to be synchronized;

[0016] Read the paging data to be synchronized one by one and execute the corresponding query statements:

[0017] If the query is successful, the query results are persistently stored and the information is routed to the next node. Subsequently, the currently processed paging data to be synchronized is deleted.

[0018] If the query fails, the query results will not be output, and the paging data to be synchronized will not be deleted;

[0019] Loop through the read and query statements until all the paging data to be synchronized is processed.

[0020] Preferably, the paging data to be synchronized is persisted in a file system, a database system or a message queue;

[0021] After reading the paging data to be synchronized one by one and successfully executing the corresponding query statement, the query results are persisted to the file system, database system or message queue.

[0022] In a second aspect, the present invention provides a structured data synchronization system based on paging and data persistence. The technical solution adopted to solve the above technical problems is as follows:

[0023] A structured data synchronization system based on paging and data persistence, comprising:

[0024] Method building module, used to build paging methods based on time fields and paging methods based on database paging;

[0025] The paging processing module is used to obtain the data to be synchronized, select the paging method, set the paging parameters, and execute the corresponding query statement to obtain the paging data to be synchronized;

[0026] A cache execution module is used to cache the paging data to be synchronized, execute corresponding query statements on the paging data to be synchronized, and cache them;

[0027] The breakpoint resume module is used to continue executing the paging data to be synchronized that was not completed last time based on the cached query results after the system is restarted or a failure occurs, thereby realizing breakpoint resume.

[0028] Optionally, when the data to be synchronized contains an incremental timestamp or date field, the method construction module constructs a paging method based on the time field, and the paging method based on the time field includes four paging parameters: time range, data volume per page, time period segmentation, and query statement;

[0029] When there is no incremental field in the data to be synchronized, the method construction module constructs a paging method based on database paging, and the paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset and query statement.

[0030] Optionally, the cache execution module involved specifically includes:

[0031] A paging cache unit is used to persistently store the paging data to be synchronized;

[0032] The read execution unit is used to read the paging data to be synchronized one by one and execute the corresponding query statements:

[0033] A judgment processing unit is used to judge whether the query operation is successful, and persist the query result to the paging cache unit when the query is successful, and not output the query result when the query fails, and not delete the current paging data to be synchronized;

[0034] The transfer deletion unit is used to route the information to the next node when the query is successful and delete the currently processed paging data to be synchronized;

[0035] The loop execution unit is used to determine whether there is paging data to be synchronized in the paging cache unit. If yes, the read execution unit is called; if no, the execution is terminated.

[0036] Preferably, the paging cache unit involved is a file system, a database system or a message queue.

[0037] Compared with the prior art, the structured data synchronization method and system based on paging and data persistence of the present invention have the following beneficial effects:

[0038] 1. The present invention can utilize limited resources to synchronize data, and after an abnormality occurs in the data synchronization process, it can quickly recover and realize breakpoint resumption, thereby improving the performance and stability of data synchronization, and improving the stability and automation efficiency of the system;

[0039] 2. The present invention can save the data to a file system or other storage medium by persistently storing the paging data to be synchronized, so that the data can be accessed and processed when needed, which helps to improve the reliability and accessibility of the data;

[0040] 3. The present invention provides two paging methods, namely, a paging method based on a time field and a paging method based on a database. According to different data characteristics and requirements, a suitable paging method can be selected to optimize the data processing process and improve synchronization efficiency;

[0041] 4. The present invention can flexibly select paging methods and set paging parameters according to actual needs, and adapt to data synchronization tasks of different scales and complexities;

[0042] 5. When the system is restarted or fails, the present invention can continue to execute the paging data to be synchronized that was not completed last time based on the cached query results, realize breakpoint resume, ensure data integrity and consistency, and avoid data loss due to system failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0044] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0046] Embodiment 1:

[0047] Combined with Figure 1 This embodiment proposes a structured data synchronization method based on paging and data persistence, which includes the following steps:

[0048] S1. Construct a paging method based on time fields and a paging method based on database paging, including:

[0049] When the data to be synchronized contains an incremental timestamp or date field, a paging method based on the time field is constructed. The paging method based on the time field includes four paging parameters: time range (minimum time and maximum time), data volume per page (for example, one page of data per day), time period segmentation (such as segmentation by day, hour, etc.), and query statement; for example, when one page of data per day, the data range of each page can be expressed as [minimum time, minimum time + 1 day), (minimum time + 1 day, minimum time + 2 days), (minimum time + 2 days, minimum time + 3 days)...(maximum time - 1, maximum time];

[0050] When there is no incremental field in the data to be synchronized, a paging method based on database paging is constructed. The paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset (used to skip processed data) and query statement.

[0051] S2. Obtain the data to be synchronized, select a paging method based on a time field or a paging method based on database paging, set corresponding paging parameters, execute corresponding query statements, and obtain the paging data to be synchronized.

[0052] S3: Cache the paging data to be synchronized, execute the corresponding query statements for the paging data to be synchronized and cache them, including:

[0053] Persist the paging data to be synchronized in the file system, database system or message queue;

[0054] Read the paging data to be synchronized one by one and execute the corresponding query statements:

[0055] If the query is successful, the query result is persisted to the file system, database system or message queue, and the information is routed to the next node. Subsequently, the currently processed paging data to be synchronized is deleted.

[0056] If the query fails, the query results will not be output, and the paging data to be synchronized will not be deleted;

[0057] Loop through the read and query statements until all the paging data to be synchronized is processed.

[0058] S4. After the system is restarted or fails, the paging data to be synchronized that was not completed last time is continued to be executed based on the cached query results to achieve breakpoint resume.

[0059] Embodiment 2:

[0060] Combined with Figure 2 This embodiment proposes a structured data synchronization system based on paging and data persistence, which includes:

[0061] Method building module, used to build paging methods based on time fields and paging methods based on database paging;

[0062] The paging processing module is used to obtain the data to be synchronized, select the paging method, set the paging parameters, and execute the corresponding query statement to obtain the paging data to be synchronized;

[0063] A cache execution module is used to cache the paging data to be synchronized, execute corresponding query statements on the paging data to be synchronized, and cache them;

[0064] The breakpoint resume module is used to continue executing the paging data to be synchronized that was not completed last time based on the cached query results after the system is restarted or a failure occurs, thereby realizing breakpoint resume.

[0065] In this embodiment, when the data to be synchronized contains an incremental timestamp or date field, a paging method based on the time field is constructed through a method construction module. The paging method based on the time field includes four paging parameters: time range (minimum time and maximum time), amount of data per page (for example, one page of data per day), time period segmentation (such as segmentation by day, hour, etc.) and query statement.

[0066] For example, when the data for each day is one page, the data range of each page can be expressed as [minimum time, minimum time + 1 day), (minimum time + 1 day, minimum time + 2 days), (minimum time + 2 days, minimum time + 3 days)...(maximum time - 1, maximum time].

[0067] In this embodiment, when there is no incremental field in the data to be synchronized, a paging method based on database paging is constructed through a method construction module. The paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset (used to skip processed data) and query statement.

[0068] In this embodiment, the cache execution module involved specifically includes:

[0069] A paging cache unit is used to persistently store the paging data to be synchronized;

[0070] The read execution unit is used to read the paging data to be synchronized one by one and execute the corresponding query statements:

[0071] A judgment processing unit is used to judge whether the query operation is successful, and persist the query result to the paging cache unit when the query is successful, and not output the query result when the query fails, and not delete the current paging data to be synchronized;

[0072] The transfer deletion unit is used to route the information to the next node when the query is successful and delete the currently processed paging data to be synchronized;

[0073] The loop execution unit is used to determine whether there is paging data to be synchronized in the paging cache unit. If yes, the read execution unit is called; if no, the execution is terminated.

[0074] The involved paging cache unit is a file system, a database system or a message queue.

[0075] In summary, the structured data synchronization method and system based on paging and data persistence of the present invention can utilize limited resources to synchronize data, and after an exception occurs in the data synchronization process, it can quickly recover and achieve breakpoint resumption, thereby improving the performance and stability of data synchronization, and improving the stability and automation efficiency of the system.

[0076] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A structured data synchronization method based on paging and data persistence, characterized in that: The steps include: S1. Construct a paging method based on time fields and a paging method based on database paging; S2. Obtain the data to be synchronized, select the paging method, set the paging parameters, execute the corresponding query statement, and obtain the paging data to be synchronized; S3: cache the paging data to be synchronized, execute the corresponding query statement on the paging data to be synchronized and cache it; S4. After the system is restarted or fails, the paging data to be synchronized that was not completed last time is continued to be executed based on the cached query results to achieve breakpoint resume.

2. The structured data synchronization method based on paging and data persistence according to claim 1 is characterized in that: The step S1 specifically includes: When the data to be synchronized contains an incremental timestamp or date field, a paging method based on the time field is constructed. The paging method based on the time field includes four paging parameters: time range, data volume per page, time period segmentation, and query statement. When there is no incremental field in the data to be synchronized, a paging method based on database paging is constructed. The paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset, and query statement.

3. The structured data synchronization method based on paging and data persistence according to claim 1 is characterized in that: The step S3 specifically includes: Persistently store the paging data to be synchronized; Read the paging data to be synchronized one by one and execute the corresponding query statements: If the query is successful, the query results are persistently stored and the information is routed to the next node. Subsequently, the currently processed paging data to be synchronized is deleted. If the query fails, the query results will not be output, and the paging data to be synchronized will not be deleted; Loop through the read and query statements until all the paging data to be synchronized is processed.

4. The structured data synchronization method based on paging and data persistence according to claim 3 is characterized in that: Persist the paging data to be synchronized in the file system, database system or message queue; After reading the paging data to be synchronized one by one and successfully executing the corresponding query statement, the query results are persisted to the file system, database system or message queue.

5. A structured data synchronization system based on paging and data persistence, characterized in that: It includes: Method building module, used to build paging methods based on time fields and paging methods based on database paging; The paging processing module is used to obtain the data to be synchronized, select the paging method, set the paging parameters, and execute the corresponding query statement to obtain the paging data to be synchronized; A cache execution module is used to cache the paging data to be synchronized, execute corresponding query statements on the paging data to be synchronized, and cache them; The breakpoint resume module is used to continue executing the paging data to be synchronized that was not completed last time based on the cached query results after the system is restarted or a failure occurs, thereby realizing breakpoint resume.

6. The structured data synchronization system based on paging and data persistence according to claim 5, characterized in that: When the data to be synchronized contains an incremental timestamp or date field, the method construction module constructs a paging method based on the time field, and the paging method based on the time field includes four paging parameters: time range, data volume per page, time period segmentation, and query statement; When there is no incremental field in the data to be synchronized, the method construction module constructs a paging method based on database paging, and the paging method based on database paging includes five paging parameters: total data volume, data volume per page, current page number, offset and query statement.

7. The structured data synchronization system based on paging and data persistence according to claim 5, characterized in that: The cache execution module specifically includes: A paging cache unit is used to persistently store the paging data to be synchronized; The read execution unit is used to read the paging data to be synchronized one by one and execute the corresponding query statements: A judgment processing unit is used to judge whether the query operation is successful, and persist the query result to the paging cache unit when the query is successful, and not output the query result when the query fails, and not delete the current paging data to be synchronized; The transfer deletion unit is used to route the information to the next node when the query is successful and delete the currently processed paging data to be synchronized; The loop execution unit is used to determine whether there is paging data to be synchronized in the paging cache unit. If yes, the read execution unit is called; if no, the execution is terminated.

8. The structured data synchronization system based on paging and data persistence according to claim 7, characterized in that: The paging cache unit is a file system, a database system or a message queue.

Images