Data synchronization method and device

Abstract

The present invention discloses a data synchronization method and device, and relates to the field of computer technology. A specific implementation of the method includes: receiving a synchronization request for source data, and creating or updating a synchronization task for the source data in a task library, scheduling and executing the task in a synchronization cycle, and setting the task state to semi-synchronous after successful execution, so as to continue scheduling and executing the task in a subsequent synchronization cycle, and in a subsequent synchronization cycle, after the task is scheduled and successfully executed in a semi-synchronous state, and no synchronization request for the source data is received in the subsequent synchronization cycle, the task state is set to synchronized. It can ensure the synchronization consistency of data between the source and the destination and the consistency between multiple destinations in a high-concurrency environment, reduce the use of distributed locks, reduce lock conflicts, reduce the number of middleware reads and writes, and ensure that data can be synchronized to middleware of different types and different throughputs.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data synchronization method and device. Background Art

[0002] After modifying the original data, the data synchronization center is notified to synchronize data by sending a change message. If the notification message contains complete data content, the data is directly written to the target data source. Otherwise, the synchronization center queries the latest version of the current data and writes it to the target data source. The above solution is difficult to ensure data consistency in a distributed environment. Data modification in a high-concurrency environment is prone to cause data version inconsistency problems. In addition, for data that changes very frequently, there are a large number of data synchronization requests, which will cause the middleware with lower throughput to be unable to keep up with the synchronization requirements of the middleware with higher throughput.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] In a high-concurrency environment, it is difficult to ensure data consistency between the source and the destination or between multiple destinations, and it will also cause middleware with low throughput to be unable to meet synchronization requirements. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data synchronization method and device, which can ensure the consistency of data synchronization between the source and the destination in a high-concurrency environment, ensure the consistency of data between multiple destinations, reduce the use of distributed locks, reduce lock conflicts, and reduce the number of middleware read and write times during the data synchronization cycle while meeting business requirements, thereby ensuring that data can be synchronized to middleware of different types and different throughputs.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data synchronization method is provided.

[0007] A data synchronization method comprises: receiving a synchronization request of source data, and creating or updating a synchronization task of the source data in a task library, including setting the state of the synchronization task of the source data to a pending synchronization state, and each synchronization request of the same source data corresponds to the same synchronization task; scheduling and executing the synchronization task of the source data in a synchronization cycle, and after successful execution, setting the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle, and in the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0008] Optionally, the creating or updating the synchronization task of the source data in the task library includes: creating or updating the synchronization task of the source data in the task library according to an identifier of the source data in the synchronization request of the source data.

[0009] Optionally, the method further includes: filtering synchronization requests for the source data within a filtering period.

[0010] Optionally, the method further includes: judging whether the synchronization request of the source data is within the filtering period by judging whether the identifier of the source data already exists in the storage middleware.

[0011] Optionally, it also includes: delaying a preset delay scheduling time to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period.

[0012] Optionally, it also includes: configuring the synchronization period, the delay scheduling time and the filtering period according to the system load used to execute the synchronization task.

[0013] Optionally, when executing the synchronization task of the source data, it is checked whether the version number of the synchronization task of the source data has changed. When there is no change, the synchronization task of the source data is executed successfully.

[0014] Optionally, the synchronization task of the source data is scheduled and executed at most once in each synchronization cycle, and the status of the synchronization task of the source data is updated after the execution is completed.

[0015] According to another aspect of an embodiment of the present invention, a data synchronization device is provided.

[0016] A data synchronization device includes: a synchronization task management module, which is used to receive synchronization requests of source data and create or update synchronization tasks of the source data in a task library, including setting the state of the synchronization task of the source data to a pending synchronization state, and each synchronization request of the same source data corresponds to the same synchronization task; a synchronization task scheduling and execution module, which is used to schedule and execute the synchronization task of the source data in a synchronization cycle, and after successful execution, set the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle, and in the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0017] Optionally, the synchronization task management module is further used to: create or update the synchronization task of the source data in the task library according to the identifier of the source data in the synchronization request of the source data.

[0018] Optionally, a filtering module is further included, which is used to filter the synchronization request of the source data within a filtering period.

[0019] Optionally, the synchronization task management module is further used to determine whether the synchronization request of the source data is within the filtering period by determining whether the identifier of the source data already exists in the storage middleware.

[0020] Optionally, the synchronization task scheduling execution module includes a task scheduling submodule, which is used to: delay a preset delay scheduling time to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period.

[0021] Optionally, a configuration module is further included, which is used to configure the synchronization period, the delay scheduling time and the filtering period according to the system load used to execute the synchronization task.

[0022] Optionally, the synchronization task scheduling execution module includes a task execution submodule, which is used to check whether the version number of the synchronization task of the source data has changed when executing the synchronization task of the source data. When there is no change, the synchronization task of the source data is executed successfully.

[0023] Optionally, the synchronization task of the source data is scheduled and executed at most once in each synchronization cycle, and the status of the synchronization task of the source data is updated after the execution is completed.

[0024] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0025] An electronic device includes: one or more processors; and a memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method provided in an embodiment of the present invention.

[0026] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0027] A computer readable medium stores a computer program, which, when executed by a processor, implements a data synchronization method provided by an embodiment of the present invention.

[0028] An embodiment of the above invention has the following advantages or beneficial effects: receiving a synchronization request for source data, creating or updating a synchronization task for the source data in a task library, scheduling and executing the task in a synchronization cycle, and setting the task state to semi-synchronous after successful execution, so as to continue scheduling and executing the task in a subsequent synchronization cycle, and setting the task state to synchronized after the task is scheduled and successfully executed in a semi-synchronous state in a subsequent synchronization cycle, and when no synchronization request for the source data is received in the subsequent synchronization cycle. . It can ensure the consistency of data synchronization between the source and the destination in a high-concurrency environment, ensure the consistency of data between multiple destinations, reduce the use of distributed locks, reduce lock conflicts, merge synchronization operations as much as possible within the data synchronization cycle while meeting business requirements, reduce the number of middleware read and write times, and ensure that data can be synchronized to middleware of different types and different throughputs.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 is a schematic diagram of main steps of a data synchronization method according to an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of a system structure according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of data synchronization according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of a synchronization task design according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of a task execution flow according to an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of a flood prevention filtering process according to an embodiment of the present invention;

[0037] Figure 7 is a schematic diagram of a task scheduling execution mechanism according to an embodiment of the present invention;

[0038] Figure 8 is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention;

[0039] Fig. 9is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0040] Fig.10 It is a schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0042] Figure 1 It is a schematic diagram of main steps of a data synchronization method according to an embodiment of the present invention.

[0043] like Figure 1 As shown, a data synchronization method according to an embodiment of the present invention mainly includes the following steps S101 to S102.

[0044] Step S101: receiving a synchronization request of source data, and creating or updating a synchronization task of the source data in a task library, including setting the state of the synchronization task of the source data to a pending synchronization state. Each synchronization request of the same source data corresponds to the same synchronization task.

[0045] Step S102: Schedule and execute the synchronization task of the source data in a synchronization cycle, and after successful execution, set the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle. In the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request for the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0046] Creating or updating the synchronization task of the source data in the task library may specifically include: creating or updating the synchronization task of the source data in the task library according to an identifier of the source data in the synchronization request of the source data.

[0047] This embodiment can also filter synchronization requests for source data within the filtering period.

[0048] Specifically, it can be determined whether the synchronization request of the source data is within the filtering cycle by determining whether the identifier of the source data already exists in the storage middleware.

[0049] In this embodiment, a preset delay scheduling time may be delayed to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period.

[0050] The synchronization period, delay scheduling time, and filtering period can be configured according to the system load used to perform the synchronization task.

[0051] When executing the synchronization task of the source data, you can also check whether the version number of the synchronization task of the source data has changed. When there is no change, the synchronization task of the source data is executed successfully.

[0052] The source data synchronization task is scheduled and executed at most once in each synchronization cycle, and the status of the source data synchronization task is updated after the execution is completed.

[0053] The data synchronization process of another embodiment of the present invention mainly includes the following steps:

[0054] Receive a synchronization request for source data, and create or update a synchronization task for the source data in the task library, including setting the state of the synchronization task for the source data to a pending synchronization state. Each synchronization request for the same source data corresponds to the same synchronization task;

[0055] If the synchronization task of the source data has not been executed in the current synchronization cycle, the synchronization task of the source data is scheduled and executed, and after successful execution, the state of the synchronization task of the source data is set to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle. If in a subsequent synchronization cycle, the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and no synchronization request for the source data is received in the synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0056] The source data refers to the data that needs to be synchronized. The source data is located in the first data source. When the source data is modified or changed, it needs to be synchronized to the second data source. The first data source is the source of the source data, and the second data source is a destination of the source data. Data synchronization is achieved by executing synchronization tasks. Synchronization tasks are created, saved, and updated in the task library.

[0057] The synchronization cycle is a pre-set time period. If the synchronization task (referred to as the task) does not complete the data synchronization of the source data within a synchronization cycle, it enters the next synchronization cycle. A synchronization task is scheduled and executed at most once within a synchronization cycle. In previous synchronization tasks, there are generally only two states: pending synchronization (pending execution) and synchronized (executed). This application introduces a new state between these two states: semi-synchronization (semi-execution) state. The following is a brief introduction to the three states.

[0058] Pending synchronization: When a task is in the Pending synchronization state, it can be scheduled for execution immediately, for example, data synchronization can be performed immediately. After the task in the Pending synchronization state is executed, the next state it enters is the Semi-synchronous state.

[0059] Semi-synchronous: Tasks in the semi-synchronous state cannot be scheduled immediately and must be delayed.

[0060] Synchronized: The data has been synchronized and there are no new change requests.

[0061] The two states of pending synchronization and semi-synchronization can be converted to each other. The above three states will be further introduced in detail below.

[0062] A synchronization task in the pending synchronization state may also be referred to as a pending synchronization task, and a synchronization task in the semi-synchronous state may also be referred to as a semi-synchronous task.

[0063] In one embodiment, after receiving a synchronization request for source data and before creating or updating a synchronization task for source data in a task library, it also includes: obtaining an identifier of the source data from the synchronization request for source data, and determining whether the identifier of the source data already exists in the storage middleware.

[0064] The identifier of the source data is used to uniquely identify the source data. For example, the identifier of the source data about a product may be the SKU ID (stock keeping unit identifier, such as the SKU code) of the product; the identifier of the source data about an order may be the order number; the identifier of the source data about a shopping cart may be the user PIN (Personal Identification Number), and so on.

[0065] Storage middleware such as Redis (Remote Dictionary Server) and other middleware.

[0066] Creating or updating a synchronization task for source data in a task library may include: when there is no identification of the source data in the storage middleware, searching in the task library whether a synchronization task for the source data already exists based on the identification of the source data obtained from the synchronization request for the source data; if so, updating the synchronization task for the source data; otherwise, creating the synchronization task for the source data.

[0067] In one embodiment, when the identification of the source data does not exist in the storage middleware, after updating or creating a synchronization task for the source data, the identification of the source data can also be used as the key of the storage middleware, the identification of the source data can be written into the storage middleware, and the expiration time of the key can be set.

[0068] The semi-synchronous state of the task introduced in the embodiment of the present invention has been introduced above. The task in the semi-synchronous state cannot be scheduled immediately and must be delayed. Therefore, it is necessary to delay a preset delay scheduling time to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period, and the filtering period is determined according to the expiration time of the key identifying the source data in the storage middleware.

[0069] In one embodiment, the synchronization period, delay scheduling time, and key expiration time may be configured according to the amount of system load used to execute the synchronization task.

[0070] In one embodiment, when the identification of the source data exists in the storage middleware, the received synchronization request of the source data is filtered so as not to execute the operation of creating or updating the synchronization task of the source data in the task library.

[0071] In one embodiment, creating a synchronization task of source data further includes: setting the version number of the synchronization task of source data to an initial version number. And updating the synchronization task of source data further includes: incrementing the version number of the synchronization task of source data by a preset value. The initial version number can be preset to a value, and the preset value of the version number increment is usually set to 1, which is not limited in the present invention.

[0072] In one embodiment, executing a synchronization task of source data includes: reading the source data from a first data source and writing it into a second data source, wherein when writing into the second data source, checking whether the version number of the synchronization task of the source data that performs the write operation has changed, if there is no change, the synchronization task of the source data is executed successfully, if there is a change, the synchronization task of the source data fails to execute.

[0073] As mentioned above, a synchronization task is scheduled and executed at most once in a synchronization cycle. Therefore, the synchronization task of the source data is scheduled and executed at most once in each synchronization cycle. If the synchronization task of the source data has been executed in a synchronization cycle, but the data synchronization of the source data has not been completed, it will continue to be scheduled and executed in the next synchronization cycle of the synchronization cycle.

[0074] In one embodiment, if in a subsequent synchronization cycle, the synchronization task of the source data is scheduled and successfully executed in a semi-synchronous state, and then a synchronization request for the source data is received again in the synchronization cycle, the process returns to the step of creating or updating the synchronization task of the source data in the task library.

[0075] The embodiments of the present invention merge synchronization requests so that a high-throughput data source can synchronize data with a low-throughput middleware, and delay redundant synchronization in a semi-synchronous mode to ensure the synchronization consistency of distributed data in a high-concurrency environment, so that the semi-synchronous mechanism can complement the update operations that may be discarded through a delayed execution mechanism, and can also flexibly configure parameters such as the filtering cycle, synchronization cycle, and delay scheduling time according to the system load.

[0076] Figure 2 is a schematic diagram of a system structure according to an embodiment of the present invention.

[0077] like Figure 2 As shown, the first data source is the source of the source data of the embodiment of the present invention, and the second data source is a destination to be synchronized with the source data of the embodiment of the present invention. The first data source can be MySQL (a relational database management system), Elastic Search (a search server based on Lucene), Hbase (a distributed, column-oriented open source database), etc., and the second data source can be Elastic Search, Hbase, Redis, etc. The data synchronization method of the embodiment of the present invention is executed in the synchronization center, which has functions such as change request merging (change request is also called synchronization request or data change notification) and task scheduling.

[0078] In a certain period of time, the same data may be changed very frequently, and the synchronization center will receive many synchronization requests, generating a large number of synchronization tasks, resulting in a backlog of synchronization tasks; middleware with relatively low throughput (such as ES (Elastic Search) and Hbase) cannot meet the synchronization requests of data sources with high throughput (such as Redis). In the embodiment of the present invention, within a synchronization cycle, the changed data (source data) should have only one synchronization operation, and synchronization more than once should be regarded as redundant and unnecessary synchronization operations.

[0079] In order to solve the synchronization problems caused by frequent data changes and reduce unnecessary redundant synchronization operations, an embodiment of the present invention merges synchronization tasks, that is, merges multiple change requests for the same data within a synchronization cycle to reduce the pressure on the synchronization center system and ensure that the synchronization center can support middleware devices with various types of throughput for data synchronization.

[0080] Figure 3 FIG. 1 is a schematic diagram of data synchronization according to an embodiment of the present invention. Figure 3 As shown, data change notification: w1, w2, w3, w4, w5, ..., w x 、w x+1 ;

[0081] The synchronization period is m (m is configurable), and the current time is t0. Then:

[0082] The first synchronization time is: t1 = t0 + m

[0083] The second synchronization time is: t2 = t1 + m = (t0 + m) + m = t0 + 2m

[0084] The third synchronization time is: t3 = t2 + m = (t0 + 2m) + m = t0 + 3m

[0085] …

[0086] The nth synchronization time is: t n =t0+nm

[0087] The n+1th synchronization time is: t n+1 =t0+(n+1)m

[0088] In a synchronization cycle m, the source data will have multiple change operations, which will trigger multiple data change notifications w. No matter how many times the source data changes, it only needs to be synchronized once in a synchronization cycle, that is: synchronization is performed according to the synchronization cycle, and the number of times the synchronization task of the source data is scheduled to be executed in a synchronization cycle will not exceed once. If the synchronization task of the source data has been executed in a synchronization cycle, but the data synchronization has not been completed, it will continue to be scheduled and executed in the next synchronization cycle of the synchronization cycle. For example, assuming that a synchronization task of a source data has been executed in the synchronization cycle from t0 to t1, but the data synchronization has not been completed, it will continue to be scheduled and executed in the synchronization cycle from t1 to t2.

[0089] The following describes the synchronization task design of an embodiment of the present invention. Figure 4 is a schematic diagram of a synchronization task design according to an embodiment of the present invention, such as Figure 4 ,The properties of the synchronization task include but are not limited to the ,task number, status and version.

[0090] The change request of source data is stored as a synchronization task (referred to as task), and the data synchronization operation is completed through periodic task scheduling. In the embodiment of the present invention, the task status includes: waiting for synchronization (status = 0), semi-synchronization (status = 1), and synchronization (status = 2).

[0091] A synchronization task has at least one task version. A task may receive a change request while being executed. If version control is not performed, the task will be changed to the next state after the task is completed, resulting in the loss of the update request. Since the embodiment of the present invention sets a version for the task, when executing the synchronization task of the source data, the source data is read from the first data source and written to the second data source. When writing to the second data source, the server of the synchronization center can check whether the version number of the synchronization task of the source data performing the write operation has changed. If there is no change, the synchronization task is executed successfully. If there is a change, the synchronization task fails.

[0092] Figure 5 FIG. 1 is a schematic diagram of a task execution process according to an embodiment of the present invention. Figure 5 As shown, in one embodiment, after receiving a change request (change notification) ① for source data, flood prevention filtering ② is performed (flood prevention filtering will be described in detail below). If the change request is filtered out (i.e., ② in the figure corresponds to the "yes" branch), the process ends; if the change request is not filtered out (i.e., ② in the figure corresponds to the "no" branch), it is determined whether the corresponding task exists in the task library ③. If the task exists, the task is updated ④, that is, the task status is updated to pending synchronization (status = 0), and the task version is incremented (to prevent the status from being updated to completed after the task being executed at this time is completed, resulting in the loss of the change request); if the task does not exist, a new task to be synchronized (i.e., a task is created) ⑤, and then it is determined whether the task ⑥ is executed synchronously. If it is executed synchronously, data synchronization ⑦ is executed immediately. If it is not executed synchronously, the current process ends.

[0093] from Figure 5 As can be seen from the process, the embodiment of the present invention will only create a synchronization task for the source data if the synchronization task for the source data does not exist. If the synchronization task for the source data already exists, only the synchronization task for the source data will be incremented. Therefore, for multiple change requests for the same source data, there is at most one synchronization task in the task library. The process of incrementing the task version is the process of task merging (such as Figure 5 (shown in the dashed box).

[0094] In addition, the execution of synchronization tasks can be done synchronously or asynchronously. When synchronous execution is selected, data synchronization starts immediately; when asynchronous execution is selected, the task enters the asynchronous waiting queue and is executed through the task scheduling mechanism.

[0095] The flood prevention and filtering mechanism of an embodiment of the present invention is described in detail below. Since the number of change requests may be so large that the underlying task library cannot satisfy the requests, an embodiment of the present invention introduces a flood prevention and filtering mechanism to eliminate the peak. The flood prevention and filtering mechanism can be implemented with the help of Redis, which also has a high throughput, and filtering is performed by determining whether a specified key exists.

[0096] Figure 6 FIG. 1 is a schematic diagram of a flood prevention filtering process according to an embodiment of the present invention. Figure 6 As shown in the figure, after receiving the synchronization request (i.e. update request) of the source data, it is first determined whether Redis has a business key (key) that identifies the source data. If the key does not exist, the key is set in Redis. The key is the identifier of the source data, and the expiration time of the key is specified as n (n represents a period of time). n is the filtering period. At this time, false is returned, indicating that the synchronization request has not been filtered, and then the subsequent steps of task merging are executed, i.e. Figure 5 Steps ③ to ⑦; if the key exists, it means that the current update request is within the filtering cycle, then true is returned to filter out the synchronization request, and no data synchronization notification is required again, that is, the operation of creating or updating the synchronization task of the source data in the task library is not executed. After the filtering cycle is introduced, the update request in the last filtering cycle will be lost, so the semi-synchronous mode must be introduced during task scheduling to make up for the lost update request by delayed execution.

[0097] The semi-synchronous mode is a mode in which tasks are scheduled in a semi-synchronous state. It has been introduced above that there are three types of task states in the embodiment of the present invention: pending synchronization, semi-synchronous, and synchronized. Among them, when a task is in the pending synchronization state, it can be scheduled for execution immediately; a task in the semi-synchronous state cannot be scheduled immediately and must be delayed. The purpose of the delay is to make up for the update requests that may be discarded within the filtering cycle n. Therefore, it must be delayed for a delayed scheduling time y (y represents a preset duration, and y>n) before scheduling and execution. The delayed scheduling time must be greater than the filtering cycle.

[0098] The embodiment of the present invention adopts a semi-synchronous strategy to ensure the final consistency of data by redundantly performing a synchronization operation at the end. It can be used in most data synchronization scenarios and even in these scenarios, the distributed lock can be removed to reduce lock conflicts in a high-concurrency environment.

[0099] When the synchronization center is scheduling tasks, it needs to find tasks that are in the "pending synchronization" state or the "semi-synchronization" state, and whose last execution time has exceeded the "filter period" from the current state, and schedule and execute them. When updating tasks after task execution, the tasks will enter the next state under the premise of satisfying version control.

[0100] The following is an introduction to the data synchronization process of an embodiment of the present invention in combination with the above embodiments. After the first data source changes the source data, it sends a synchronization request for the source data to the synchronization center. After receiving the synchronization request for the source data, the synchronization center searches for the key that identifies the source data in Redis. If the key exists in Redis, it means that it is still within the filtering cycle of the key (that is, the key exists and has not expired), and the synchronization request is filtered without being stored in the database; if the key does not exist in Redis, it is further determined whether there is a synchronization task corresponding to the source data in the task library.

[0101] If there is a synchronization task for the source data in the task library, the synchronization task is updated, and the update operation includes: increasing the version number of the task by 1, and setting the status of the task to 0 (i.e., setting it to pending synchronization); if there is no synchronization task for the source data in the task library, the synchronization task is created, and the creation operation includes: setting the version number of the task to the initial version number, and setting the status of the task to 0 (i.e., setting it to pending synchronization).

[0102] In the case where there is a synchronization task of the source data in the task library, there are currently three possible situations for the synchronization task: Situation 1: it may not have been executed in the current synchronization cycle, and the task status is 0 (to be synchronized); Situation 2: it has been executed in the current synchronization cycle, and the task status is 1 (semi-synchronized), but the synchronization task will not be scheduled and executed again in this synchronization cycle (because a synchronization task can only be executed once in a synchronization cycle), and can only be scheduled and executed in the next synchronization cycle; Situation 3: it has been executed in the current synchronization cycle, and the task status is 0 (to be synchronized). For example, in the case of situation 3, the task is scheduled and successfully executed in the state to be synchronized in the current synchronization cycle, and the task is set to the semi-synchronized state. Afterwards, the synchronization request of the source data is received again in the synchronization cycle, and the update of the synchronization task is executed, that is, the state of the task is set to the state to be synchronized again. Therefore, the two states of the synchronization task to be synchronized and semi-synchronized in the embodiment of the present invention can be converted to each other.

[0103] For the above situation 2, the task status is updated, that is, the task status is set from 1 to 0 (the semi-synchronous state is converted to the waiting synchronization state); for the above situation 1 and situation 2, the task status is updated, that is, the status is set from 0 to 0 (maintaining the waiting synchronization state);

[0104] After creating or updating a task, if the synchronization task of the source data has been executed in the current synchronization cycle, it needs to wait for the next synchronization cycle to schedule execution. If it has not been executed in the current synchronization cycle, it will be scheduled for execution in the current synchronization cycle, and the task status will be set to 1 (semi-synchronous state) after execution.

[0105] According to the above introduction, it can be known that the task status of the source data entering the next synchronization cycle of the above current synchronization cycle may be to be synchronized or semi-synchronized. For the convenience of description, the above current synchronization cycle is recorded as cycle I, and the next synchronization cycle of the above current synchronization cycle is recorded as cycle II. Among them, if the status of the task entering cycle II is to be synchronized, then after the task is scheduled and executed successfully in cycle II, the task status is set to semi-synchronized; if the status of the task entering cycle II is semi-synchronized, the scheduling and execution of the semi-synchronous task is delayed, so that the time difference between the time of scheduling and executing the semi-synchronous task in cycle II and the time when the task is set to the semi-synchronous state is greater than the filtering cycle n. If a new synchronization request for the source data is received after the task is set to semi-synchronous state, and after the time difference from the time when the task is set to the semi-synchronous state is equal to the time point of filtering cycle n, and before the task to be synchronized is scheduled and executed in this cycle II, then at this time, due to the end of the filtering cycle, the key identified by the source data in Redis has expired and been cleared, then the key identified by the source data does not exist in Redis, then the source data identifier is used as the key and written to Redis, and since the semi-synchronous task can be found in the task library according to the identifier of the source data, then the task is updated, that is, the task status is set from semi-synchronous to pending synchronization, and the task version number is increased by 1, and then after the task is scheduled in cycle II, the task status is set from pending synchronization to semi-synchronous, and since the task has been executed once in cycle II, but the data synchronization has not been completed, then it will continue to be scheduled and executed in the next synchronization cycle of cycle II (recorded as cycle III), and so on.

[0106] If within a certain synchronization cycle X, after the semi-synchronous task scheduling is executed and until the end of the synchronization cycle X, the synchronization center does not receive any synchronization request for the source data, then after the end of the cycle X, the task status is set to 2 (synchronized) to complete the entire data synchronization process.

[0107] If within a certain synchronization cycle Y, after the semi-synchronous task is executed and before the end of the synchronization cycle Y, the synchronization center receives a synchronization request for the source data, referring to the above introduction, the state of the semi-synchronous task will become the pending synchronization state, and it will be scheduled for execution in the next synchronization cycle of the synchronization cycle Y. The situation in which the state of the task entering the synchronization cycle is pending synchronization has been introduced above and will not be repeated here.

[0108] When the synchronization task of the embodiment of the present invention is executed, the latest source data is pulled from the first data source (the source of the source data) for synchronization. Since the semi-synchronous task of the embodiment of the present invention is delayed in scheduling and execution, and the delayed scheduling time is greater than the filtering cycle, the semi-synchronous task must be scheduled for execution outside the filtering cycle. Therefore, even if some synchronization requests are filtered out in the most recent filtering cycle, since the semi-synchronous task will pull the latest source data from the first data source for synchronization when it is scheduled for execution, the filtered part of the synchronization requests will also be supplemented, that is, the data synchronized to the second data source (a destination to be synchronized) is still the latest data, and there is no data loss.

[0109] Figure 7 FIG. 1 is a schematic diagram of a task scheduling execution mechanism according to an embodiment of the present invention. Figure 7 As shown, after the task to be synchronized in the embodiment of the present invention performs the first synchronization operation, the state changes to the semi-synchronous state, and after the semi-synchronous task performs the second operation, the state changes to the synchronized state. Since the same task can only be scheduled and executed once in a synchronization cycle, the first synchronization operation (corresponding to Figure 7 The first synchronization) and the second synchronization operation (corresponding to Figure 7 The second synchronization of the synchronization task is executed in different synchronization cycles. In addition, the task to be synchronized can be executed immediately, while the semi-synchronous task needs to be executed with a delay. This has been introduced above and will not be repeated here.

[0110] Figure 8 4 is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention.

[0111] like Figure 8 As shown, a data synchronization device 800 according to an embodiment of the present invention mainly includes: a synchronization task management module 801 and a synchronization task scheduling execution module 802 .

[0112] The synchronization task management module 801 is used to receive synchronization requests for source data and create or update synchronization tasks for the source data in the task library, including setting the state of the synchronization task of the source data to a pending synchronization state. Each synchronization request for the same source data corresponds to the same synchronization task.

[0113] The synchronization task scheduling and execution module 802 is used to schedule and execute the synchronization task of the source data in a synchronization cycle, and after successful execution, set the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle. In the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0114] The synchronization task management module 801 may be specifically configured to create or update the synchronization task of the source data in the task library according to the identifier of the source data in the synchronization request of the source data.

[0115] The data synchronization device 800 may further include a filtering module, which is used to filter the synchronization request of the source data within a filtering period.

[0116] The synchronization task management module 801 can be specifically used to determine whether the synchronization request of the source data is within the filtering period by determining whether the identifier of the source data already exists in the storage middleware.

[0117] The synchronization task scheduling execution module 802 may include a task scheduling submodule, which is used to: delay a preset delay scheduling time to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period.

[0118] The data synchronization device 800 may further include a configuration module, which is used to configure the synchronization period, the delay scheduling time and the filtering period according to the system load used to execute the synchronization task.

[0119] The synchronization task scheduling execution module 802 may include a task execution submodule, which is used to check whether the version number of the synchronization task of the source data has changed when executing the synchronization task of the source data. When there is no change, the synchronization task of the source data is executed successfully.

[0120] The synchronization task of the source data is scheduled and executed at most once in each synchronization cycle, and the status of the synchronization task of the source data is updated after the execution is completed.

[0121] A data synchronization device according to another embodiment of the present invention includes: a synchronization task management module, a synchronization task scheduling execution module, a filtering module, and a configuration module.

[0122] The synchronization task management module is used to receive synchronization requests for source data and create or update synchronization tasks for source data in the task library, including setting the state of the synchronization task of the source data to the pending synchronization state. Each synchronization request for the same source data corresponds to the same synchronization task. The synchronization task scheduling and execution module is used to schedule and execute the synchronization task of the source data if the synchronization task of the source data has not been executed in the current synchronization cycle, and after successful execution, set the state of the synchronization task of the source data to the semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in the subsequent synchronization cycle. If in a subsequent synchronization cycle, the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and no synchronization request for the source data is received in the synchronization cycle, the state of the synchronization task of the source data is set to the synchronized state, thereby completing the data synchronization of the source data.

[0123] The filtering module can be specifically used to obtain the identifier of the source data from the synchronization request of the source data, and determine whether the identifier of the source data already exists in the storage middleware.

[0124] The synchronization task management module can be specifically used for: when there is no identification of the source data in the storage middleware, based on the identification of the source data obtained from the synchronization request of the source data, searching in the task library whether a synchronization task of the source data already exists; if so, updating the synchronization task of the source data; otherwise, creating a synchronization task of the source data.

[0125] When the source data identifier does not exist in the storage middleware, after the synchronization task management module updates or creates a synchronization task for the source data, the filtering module is also used to: use the source data identifier as the key of the storage middleware, write the source data identifier into the storage middleware, and set the expiration time of the key.

[0126] The filtering module is also used to: when there is an identifier of the source data in the storage middleware, filter the received synchronization request of the source data so as not to execute the operation of creating or updating the synchronization task of the source data in the task library.

[0127] The synchronization task scheduling execution module may include a task scheduling submodule, which is used to: delay a preset delay scheduling time to schedule the synchronization task of the source data in the semi-synchronous state, wherein the delay scheduling time is greater than the filtering period, and the filtering period is determined according to the expiration time of the key identifying the source data in the storage middleware.

[0128] The configuration module is used to configure the synchronization period, the delay scheduling time and the key expiration time according to the system load quantity used to execute the synchronization task.

[0129] Creating a synchronization task of the source data may further include: setting the version number of the synchronization task of the source data to an initial version number.

[0130] Updating the synchronization task of the source data may further include: increasing the version number of the synchronization task of the source data by a preset value, for example, by 1.

[0131] The synchronization task scheduling execution module also includes a task execution sub-module, which is used to: read the source data from the first data source and write it into the second data source, wherein when writing to the second data source, check whether the version number of the synchronization task of the source data that performs the write operation has changed. If there is no change, the synchronization task of the source data is executed successfully. If there is a change, the synchronization task of the source data fails to execute.

[0132] The synchronization task of the source data is scheduled and executed at most once in each synchronization cycle. If the synchronization task of the source data has been executed in a synchronization cycle but the data synchronization has not been completed, it will continue to be scheduled and executed in the next synchronization cycle of the synchronization cycle.

[0133] The synchronization task scheduling and execution module is also used for: if in a subsequent synchronization cycle, the synchronization task of the source data is scheduled and executed successfully in a semi-synchronous state, and a synchronization request for the source data is received again in the synchronization cycle, then return to the above-mentioned step of creating or updating the synchronization task of the source data in the task library.

[0134] In addition, the specific implementation content of the data synchronization device in the embodiment of the present invention has been described in detail in the data synchronization method described above, so the repeated content will not be described again here.

[0135] Fig. 9 An exemplary system architecture 900 is shown to which the data synchronization method or data synchronization device according to the embodiment of the present invention can be applied.

[0136] like Fig. 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, network 904 and server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0137] Users can use terminal devices 901, 902, 903 to interact with server 905 through network 904 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 901, 902, 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0138] The terminal devices 901 , 902 , and 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0139] The server 905 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 901, 902, and 903. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0140] It should be noted that the data synchronization method provided in the embodiment of the present invention is generally executed by the server 905 , and accordingly, the data synchronization device is generally arranged in the server 905 .

[0141] It should be understood that Fig. 9 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0142] Reference below Fig.10 , which shows a structural diagram of a computer system 1000 suitable for implementing a server of an embodiment of the present application. Fig.10 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0143] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0144] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0145] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the main step schematic diagram can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the main step schematic diagram. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present application are executed.

[0146] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0147] The main step schematic diagram and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the main step schematic diagram or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or the main step schematic diagram, and the combination of the boxes in the block diagram or the main step schematic diagram can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor. For example, they may be described as follows: a processor includes a synchronization task management module and a synchronization task scheduling execution module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the synchronization task management module may also be described as "a module for receiving synchronization requests for source data and creating or updating synchronization tasks for source data in a task library."

[0149] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving a synchronization request of source data, and creating or updating a synchronization task of the source data in a task library, including setting the state of the synchronization task of the source data to a pending synchronization state, and each synchronization request of the same source data corresponds to the same synchronization task; scheduling and executing the synchronization task of the source data in a synchronization cycle, and after successful execution, setting the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle, and in the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data.

[0150] According to the technical solution of the embodiment of the present invention, a synchronization request of source data is received, and a synchronization task of the source data is created or updated in the task library. The task is scheduled and executed in a synchronization cycle, and the task state is set to semi-synchronous after successful execution, so as to continue to schedule and execute the task in a subsequent synchronization cycle. In a subsequent synchronization cycle, after the task is scheduled and successfully executed in a semi-synchronous state, and no synchronization request of the source data is received in the subsequent synchronization cycle, the task state is set to synchronized, thereby completing the data synchronization of the source data. It is possible to ensure the consistency of data synchronization between the source and the destination in a high-concurrency environment, ensure the consistency of data between multiple destinations, reduce the use of distributed locks, reduce lock conflicts, and merge synchronization operations as much as possible within the data synchronization cycle while meeting business requirements, reduce the number of middleware read and write times, and ensure that data can be synchronized to middleware of different types and different throughputs.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Receive a synchronization request for source data, and create or update a synchronization task for the source data in a task library, including setting the state of the synchronization task for the source data to a pending synchronization state, wherein each synchronization request for the same source data corresponds to the same synchronization task; The synchronization task of the source data is scheduled and executed in a synchronization cycle, and after successful execution, the state of the synchronization task of the source data is set to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle. In the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data; The synchronization cycle is a pre-set time period. If the synchronization task fails to complete the data synchronization of the source data within a synchronization cycle, it enters the next synchronization cycle. Tasks in a semi-synchronous state are scheduled after delayed scheduling.

2. The method according to claim 1, characterized in that The step of creating or updating the synchronization task of the source data in the task library includes: A synchronization task for the source data is created or updated in a task library according to an identifier of the source data in the synchronization request for the source data.

3. The method according to claim 1, characterized in that Also includes: The synchronization request of the source data is filtered within the filtering period, and whether the synchronization request of the source data is within the filtering period is determined by determining whether the identifier of the source data already exists in the storage middleware.

4. The method according to claim 3, characterized in that Also includes: The synchronization task of the source data in the semi-synchronous state is scheduled by delaying a preset delay scheduling time, wherein the delay scheduling time is greater than the filtering period.

5. The method according to claim 4, characterized in that Also includes: The synchronization period, the delay scheduling time and the filtering period are configured according to the system load for executing the synchronization task.

6. The method according to claim 1, characterized in that When executing the synchronization task of the source data, check whether the version number of the synchronization task of the source data has changed. When there is no change, the synchronization task of the source data is executed successfully.

7. The method according to claim 1, characterized in that The synchronization task of the source data is scheduled and executed at most once in each synchronization cycle, and the status of the synchronization task of the source data is updated after the execution is completed.

8. A data synchronization device, characterized in that: include: A synchronization task management module, used to receive synchronization requests for source data, and create or update synchronization tasks for the source data in a task library, including setting the state of the synchronization task for the source data to a pending synchronization state, and each synchronization request for the same source data corresponds to the same synchronization task; A synchronization task scheduling and execution module, used for scheduling and executing the synchronization task of the source data in a synchronization cycle, and after successful execution, setting the state of the synchronization task of the source data to a semi-synchronous state, so as to continue to schedule and execute the synchronization task of the source data in a subsequent synchronization cycle of the synchronization cycle, and in the subsequent synchronization cycle, after the synchronization task of the source data is scheduled and successfully executed in the semi-synchronous state, and when no synchronization request of the source data is received in the subsequent synchronization cycle, the state of the synchronization task of the source data is set to a synchronized state, thereby completing the data synchronization of the source data; The synchronization cycle is a pre-set time period. If the synchronization task fails to complete the data synchronization of the source data within a synchronization cycle, it enters the next synchronization cycle. Tasks in a semi-synchronous state are scheduled after delayed scheduling.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images