Data processing method, electronic equipment and storage medium

Abstract

The invention provides a data processing method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a data bifurcation point of a source cluster and a target cluster, parallelly judging whether a plurality of log serial numbers in front of the data bifurcation point in the target cluster correspond to a first check point or not, if yes, obtaining the information of the first check point, and searching a second check point consistent with the first check point in information in the source cluster, and recovering the data in the target cluster by taking the second check point as a starting point. By searching the check points between the data branching points, the target cluster is recovered based on the check points, all data before the data branching points do not need to be compared, and the cluster recovery efficiency is improved. And whether the log serial number is the check point or not is judged in parallel, so that the check point searching efficiency is improved, the overall processing data is reduced, and the data recovery efficiency is further improved.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data processing method, electronic device and storage medium. Background Art

[0002] In a distributed system, data divergence may occur between the source cluster and the target cluster. This means that during the data replication or synchronization process, the states of the two clusters begin to become inconsistent due to system failures, network problems, etc. When data divergence occurs, data recovery or consistency repair is usually required to ensure system reliability and data consistency.

[0003] Currently, the main method is to compare the data in the source cluster and the target cluster before the data divergence point to obtain the data consistency point, and then restore or repair the consistency of the target cluster based on the data consistency point.

[0004] However, comparing the data before the data divergence point will take a lot of time and affect the efficiency of cluster recovery. Summary of the invention

[0005] The present application provides a data processing method, an electronic device and a storage medium to improve cluster recovery efficiency.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Get the data divergence point between the source cluster and the target cluster;

[0008] Determine in parallel whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint;

[0009] If yes, obtain information of the first checkpoint;

[0010] Searching the source cluster for a second checkpoint that is consistent with information of the first checkpoint;

[0011] Taking the second checkpoint as a starting point, data in the target cluster is restored.

[0012] In a possible implementation, the concurrently determining whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint includes:

[0013] Determine in parallel through multiple threads whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint;

[0014] Each of the threads corresponds to at least one log sequence number before the data divergence point in the target cluster.

[0015] In a possible implementation, the method further includes determining in parallel by multiple threads whether multiple log sequence numbers before the data divergence point in the target cluster correspond to before the first checkpoint:

[0016] Assigning a corresponding thread number to each of the multiple threads in turn;

[0017] Determine a starting log sequence number of each thread according to the data divergence point, the log block size, and the thread number of each thread;

[0018] Determine the interval step size according to the log block size and the number of threads;

[0019] According to the starting log sequence number of each thread, the number of log sequence numbers and the interval step, a corresponding log sequence number is allocated to each thread.

[0020] In a possible implementation, searching the source cluster for a second checkpoint that is consistent with information of the first checkpoint includes:

[0021] Search the source cluster for a second checkpoint that is consistent with the hash value of the first checkpoint.

[0022] In a possible implementation, obtaining the data divergence point between the source cluster and the target cluster includes:

[0023] Get the timeline of the source cluster and the timeline of the target cluster, and get the inconsistent timelines;

[0024] A data divergence point between the source cluster and the target cluster is determined according to the inconsistent timeline.

[0025] In a possible implementation manner, the number of the first checkpoints and the number of the second checkpoints are multiple;

[0026] The recovering the data in the target cluster with the second checkpoint as the starting point includes:

[0027] Searching for a target checkpoint that is closest in time to the data divergence point from the second checkpoints;

[0028] Taking the target checkpoint as a starting point, data in the target cluster is restored.

[0029] In a possible implementation manner, after acquiring the information of the first checkpoint, the method includes:

[0030] Setting a circular linked list, the circular linked list is used to store a preset number of elements, each of the elements includes information of a first checkpoint and / or information of a second checkpoint;

[0031] Each time the information of the first checkpoint is obtained, the information of the first checkpoint is written into the circular linked list;

[0032] After writing the information of the preset number of first checkpoints in the circular linked list, controlling the thread to exit;

[0033] Searching the source cluster for a second checkpoint that is consistent with information of the first checkpoint includes:

[0034] Searching for a second checkpoint corresponding to the first checkpoint in the source cluster, and writing information of the second checkpoint into the circular linked list;

[0035] Compare whether the information of the first checkpoint and the information of the second checkpoint in the circular linked list are consistent;

[0036] If so, obtain a second checkpoint that is consistent with the information of the first checkpoint.

[0037] In a possible implementation, after writing the information of the first checkpoint into the circular linked list, the method further includes:

[0038] Mark the state of the corresponding element in the circular linked list as the first state;

[0039] After writing the information of the second checkpoint into the circular linked list, marking the state of the corresponding element in the circular linked list as the second state;

[0040] The comparing whether the information of the first checkpoint and the information of the second checkpoint in the circular linked list are consistent includes:

[0041] Compare whether the information of the first checkpoint in the element marked with the second state in the circular linked list is consistent with the information of the second checkpoint.

[0042] In a second aspect, the present application provides a data processing device, the device comprising:

[0043] A first acquisition module is used to acquire data divergence points between a source cluster and a target cluster;

[0044] A determination module, used for determining in parallel whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint;

[0045] A second acquisition module, configured to acquire information of the first checkpoint after determining the first checkpoint;

[0046] A search module, configured to search the source cluster for a second checkpoint that is consistent with information of the first checkpoint;

[0047] A recovery module is used to recover the data in the target cluster with the second checkpoint as a starting point.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0049] The memory stores computer-executable instructions;

[0050] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0053] The data processing method, electronic device, and storage medium provided by the present application, after obtaining the data divergence point of the source cluster and the target cluster, determine in parallel whether the multiple log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint. If so, obtain the information of the first checkpoint, and search for a second checkpoint in the source cluster that is consistent with the information of the first checkpoint, and use the second checkpoint as the starting point to restore the data in the target cluster. By finding the checkpoints between the data divergence points and restoring the target cluster based on the checkpoints, there is no need to compare all the data before the data divergence point, thereby improving the cluster recovery efficiency. And in parallel, determine whether the log sequence number is a checkpoint, improve the efficiency of checkpoint search, reduce the overall processing data, and further improve the data recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0055] Figure 1 A schematic diagram of a scenario of a data processing method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a data flow diagram provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of a timeline provided for an embodiment of the present application;

[0058] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the correspondence between threads and log sequence numbers provided in an embodiment of the present application;

[0060] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the structure of the data processing device provided by this application;

[0062] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] As described in the background technology, currently, the data in the source cluster and the target cluster before the data divergence point are mainly compared to obtain the data consistency point, which will consume a lot of time and affect the cluster recovery efficiency.

[0066] The present application provides a data processing method, which determines in parallel whether the log sequence number before the data divergence point is a checkpoint, so as to quickly find the checkpoint and perform data recovery based on the checkpoint, without having to compare all the data before the data divergence point, thereby improving the cluster recovery efficiency.

[0067] Figure 1 A schematic diagram of a scenario of a data processing method provided in an embodiment of the present application is shown, such as Figure 1As shown, the master device 102 is used as the source cluster and the slave device 103 is used as the target cluster. The master device 102 can be responsible for the data change, and then copy these changes to the slave device 103 to synchronize the data of the slave device 103 with the master device 102. In the process of the master device 102 copying the changes to the slave device 103, due to network instability, terminal, system failure (for example, hardware or software failure of the master device or the slave device), the data of the slave device is not synchronized with the data of the master device, and a data divergence point occurs. The electronic device 101 obtains the data divergence point of the master device 102 and the slave device 103, and determines in parallel whether the multiple log sequence numbers before the data divergence point in the slave device 103 correspond to the first checkpoint. If so, the information of the first checkpoint is obtained. Then, the electronic device 101 searches for the second checkpoint consistent with the information of the first checkpoint from the master device 102, and resynchronizes the data between the second checkpoint and the data divergence point in the master device 102 to the slave device 103, so as to achieve effective data recovery.

[0068] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0069] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, with the electronic device as the execution subject, the data processing method provided in the embodiment of the present application may include the following steps:

[0070] S201. Obtain data divergence points between a source cluster and a target cluster.

[0071] The source cluster is the starting location of the data and is responsible for providing the data that needs to be migrated or synchronized. The target cluster is the destination of the data and is responsible for receiving and storing the data transmitted from the source cluster. During the data migration or synchronization process, there may be a time or location where data inconsistency occurs between the source cluster and the target cluster, which is a data divergence point. Therefore, the electronic device can obtain the time or location where data inconsistency occurs between the source cluster and the target cluster as the data divergence point.

[0072] In some embodiments, the timeline of the source cluster and the timeline of the target cluster are obtained, and the inconsistent timeline is obtained, and the data divergence point of the source cluster and the target cluster is determined according to the inconsistent timeline. The timeline provides the sequence and time point of event occurrence. By analyzing the timeline, the starting point of data inconsistency can be accurately located, which helps to quickly identify the data divergence point.

[0073] A timeline is a sequence of events or transactions arranged in chronological order. In a database or distributed system, a timeline is used to record the order in which transactions are submitted, the order in which logs are written, etc. When problems such as data delays, network problems, system failures, and concurrent modifications occur, the timeline of the source cluster may be inconsistent with the timeline of the target cluster, resulting in inconsistent data between the source and target clusters.

[0074] For example, due to data delays caused by network latency, excessive system load, or resource limitations, the target cluster may not receive or process all data changes of the source cluster in a timely manner, resulting in inconsistency between the timeline of the target cluster and the timeline of the source cluster. Network interruption or instability may cause data packet loss or delayed transmission, resulting in inconsistency between the timeline of the source cluster and the timeline of the target cluster, resulting in inconsistent data between the source cluster and the target cluster. During the synchronization process, if the data on the source cluster is still being modified and these modifications are not synchronized to the target cluster in a timely manner, the timeline of the source cluster may be inconsistent with the timeline of the target cluster.

[0075] For example, Figure 3 As shown, the timeline of the source cluster includes timeline 1 and timeline 2. Timeline 1 of the source cluster corresponds to log sequence number Isn1-log sequence number IsnX, and timeline 2 of the source cluster corresponds to log sequence number IsnX+1-log sequence number IsnX+n. The target cluster only corresponds to timeline 1, and timeline 1 of the target cluster corresponds to log sequence numbers Isn1-IsnX and IsnA-IsnN. Therefore, the timeline of the source cluster and the timeline of the target cluster can indicate whether there is inconsistency between the source cluster and the target cluster.

[0076] Therefore, the timeline of the source cluster and the timeline of the target cluster can be compared to find the inconsistent timeline. The timeline in the source cluster that is inconsistent with the target cluster can be recorded as the first timeline, and the timeline in the target cluster that is inconsistent with the source cluster can be recorded as the second timeline. Then, the log sequence number (LSN) of the first timeline and the second timeline can be compared to determine the data divergence point. Since the log sequence number identifies the position in the transaction log, the data divergence point can be any sequence number in the inconsistent log sequence number, for example, it can be the log sequence number with the earliest position.

[0077] For example, the source cluster includes timeline 1 and timeline 2, and the target cluster only includes timeline 1. The log sequence number corresponding to timeline 1 of the source cluster can be compared with the log sequence number corresponding to timeline 1 of the target cluster, and at least part of the inconsistent log sequence numbers can be recorded as data divergence points.

[0078] S202. Determine in parallel whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint.

[0079] A checkpoint is a snapshot of the system at a certain moment, which saves the current state of the database or system, including the data and metadata of all committed transactions. In the event of a failure, data can be recovered based on the checkpoint without having to replay all transaction logs from the beginning, reducing the recovery time. By creating a checkpoint, you can clear or archive transaction logs, reduce the size of the log file, and thus improve system performance. When creating a checkpoint, you can also record the current log sequence number. The log sequence number provides a unique identifier for marking the position in the transaction log. By recording the log sequence number at the checkpoint, the system can clearly know from which position to replay the transaction log to restore to the latest state after the checkpoint. It can be understood that the checkpoint records the complete state of the system at a certain moment, including all committed data and metadata. The data divergence point may be located in the middle of a transaction when it corresponds to the log sequence number. By starting from the checkpoint, the system can ensure that all transactions are fully processed. The log sequence numbers before the data divergence point do not all correspond to the checkpoint. Therefore, after obtaining the divergence point, it is determined whether the log sequence number before the data divergence point corresponds to the first checkpoint.

[0080] In this embodiment, in order to further improve the determination efficiency, it may be determined in parallel whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint.

[0081] In some embodiments, starting from the data divergence point, multiple threads are used to determine in parallel whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint. Multiple threads can simultaneously determine whether different log sequence numbers correspond to the first checkpoint, reducing the overall processing time. Each thread can correspond to at least one log sequence number before the data divergence point in the target cluster, and each thread judges the corresponding log sequence number in turn, and the number of judgments of each thread is the number of corresponding log sequence numbers.

[0082] It should be noted that if the log sequence number corresponds to the first checkpoint, step S203 is executed. Since there may be multiple log sequence numbers corresponding to the first checkpoint before the data divergence point, that is, the number of first checkpoints is multiple, step S203 is executed each time it is determined that the log sequence number corresponds to the first checkpoint.

[0083] S203: Obtain information of the first checkpoint.

[0084] In this step, after determining that the log sequence number corresponds to the first checkpoint, the information of the first checkpoint can be obtained. The information of the first checkpoint may include a set of key information and metadata, such as the log sequence number, identifier, etc. The identifier may include a hash value, for example.

[0085] S204: Search the source cluster for a second checkpoint that has information consistent with the first checkpoint.

[0086] For example, after obtaining the first checkpoint, the second checkpoint corresponding to the first checkpoint is searched in the source cluster. The checkpoints in the source cluster and the checkpoints in the target cluster may correspond. After finding the second checkpoint in the source cluster, the information of the second checkpoint may be obtained, and then the information of the first checkpoint and the information of the second checkpoint are compared to obtain the second checkpoint whose information is consistent with the information of the first checkpoint.

[0087] In some embodiments, a second checkpoint that is consistent with the hash value of the first checkpoint is searched in the source cluster so as to obtain the second checkpoint more quickly and accurately. For example, after finding the second checkpoint corresponding to the first checkpoint in the source cluster, the hash value of the first checkpoint and the hash value of the second checkpoint are compared to obtain the second checkpoint whose hash value is consistent with the hash value of the first checkpoint.

[0088] S205: Taking the second checkpoint as the starting point, restore the data in the target cluster.

[0089] In this step, the data in the target cluster is restored starting from the second checkpoint, without having to replay all transaction logs from the beginning, which reduces the recovery time and improves the recovery efficiency. For example, the data after the second checkpoint in the source cluster can be restored to the target cluster. After the data is restored, the log sequence number in the target cluster can be Figure 3 shown.

[0090] In some embodiments, the number of first checkpoints and the number of second checkpoints are both multiple. The target checkpoint closest to the data divergence point on the timeline can be found from the multiple second checkpoints. The target checkpoint is used as the starting point to restore the data in the target cluster. While ensuring data recovery, the amount of data to be restored is reduced, thereby improving recovery efficiency.

[0091] The data processing method provided by the present application recovers the target cluster based on the checkpoints by finding the checkpoints between the data divergence points, without having to compare all the data before the data divergence points, thereby improving the cluster recovery efficiency. In addition, the method determines in parallel whether the log sequence number is a checkpoint, thereby improving the checkpoint search efficiency, reducing the overall processing data, and further improving the data recovery efficiency.

[0092] Figure 4 A flowchart of a data processing method provided by another embodiment of the present application is shown in FIG. Figure 4 As shown, the data processing method may include the following steps: determining in parallel by multiple threads whether multiple log sequence numbers before the divergence point in the target cluster correspond to before the first checkpoint.

[0093] S301. Assign a corresponding thread number to each of a plurality of threads in sequence.

[0094] For example, each thread may be numbered with an integer, starting from 1. When the number of threads is 10, the thread numbers of each thread are 1-10 in sequence.

[0095] S302: Determine the starting log sequence number of each thread according to the data divergence point, the log block size, and the thread number of each thread.

[0096] For example, taking the starting log sequence number of a thread as an example, the starting log sequence number of the thread is: the log sequence number corresponding to the data divergence point - (thread number of the thread * log block size).

[0097] When processing WAL (Write-Ahead Logging) in parallel, using the log block size to calculate the starting log sequence number of the thread can ensure that the search operation of each thread starts from the boundary of the block, ensuring the integrity and accuracy of the search operation.

[0098] It should be noted that the log sequence number is a unique identifier used to mark the position in the database transaction log. It is usually assigned in an incremental manner to ensure that each log record has a unique log sequence number. The WAL log file consists of multiple fixed-size blocks, each of which can contain one or more log records. Each log sequence number corresponds to a specific position in the WAL log. The log sequence number can be used to directly locate a specific block in the WAL log file, thereby accessing the corresponding log record. In data recovery, the log sequence number can be used to accurately find the block where the first log record that needs to be replayed is located.

[0099] S303: Determine the interval step size according to the log block size and the number of threads.

[0100] For example, the log block size * the number of threads is used as the search interval, that is, each thread searches for one block, ensuring parallel operation of the threads and avoiding repeated searches.

[0101] S304: Allocate a corresponding log sequence number to each thread according to the starting log sequence number, the number of log sequence numbers and the interval step of each thread.

[0102] Each thread makes a judgment on the log sequence number before the data divergence point, and the log sequence number corresponding to each judgment operation of each thread can be determined according to the starting log sequence number, the number of log sequence numbers and the interval step of each thread.

[0103] For example, the log sequence number corresponding to each thread may be calculated based on the starting log sequence number of each thread minus the number of log sequence numbers corresponding to each thread*(log block size*number of threads).

[0104] Take a thread as an example. The log sequence number corresponding to each thread is: data divergence point - (thread number * log block size) - number of log sequence numbers corresponding to the thread * (log block size * number of threads). Figure 5 As shown, the log sequence numbers assigned to thread 1 are Isn1 and Isn4, and the log sequence numbers assigned to thread 2 are Isn2 and Isn5.

[0105] The data processing method provided in the embodiment of the present application allocates a corresponding log sequence number to each thread through the data divergence point, log block size, and thread number, thereby ensuring that multiple threads can judge multiple log sequence numbers in parallel to determine whether they are checkpoints, thereby improving the judgment efficiency and finding the checkpoint more quickly.

[0106] Figure 6 A flowchart of a data processing method provided in another embodiment of the present application is shown as follows: Figure 6 As shown, the data processing method provided in the embodiment of the present application may include the following steps:

[0107] S401: Obtain data divergence points between a source cluster and a target cluster.

[0108] S402: Set a circular linked list.

[0109] The circular linked list is used to store a preset number of elements, each element including information of a first checkpoint and / or information of a second checkpoint.

[0110] It should be noted that the circular linked list is used to store a preset number of elements, that is, it can only store information of the preset number of first checkpoints, thereby limiting the number of log sequence number judgments, avoiding judging too many log sequence numbers, and improving data processing efficiency. The specific value of the preset number can be determined according to actual conditions and is not limited here.

[0111] For example, the number of first checkpoints may be multiple, and after finding a second checkpoint corresponding to the first checkpoint in the source cluster, the information of the second checkpoint is written into the circular linked list, and the element includes the information of the first checkpoint and the information of the second checkpoint. When the second checkpoint corresponding to the first checkpoint is not found in the source cluster, the element only includes the information of the first checkpoint.

[0112] For example, each element in the circular linked list may have a corresponding order, and each element has a corresponding next item pointer, which is used to indicate the next element. If the current item is the last element, the next item pointer may point to the first element of the circular linked list. This arrangement can avoid missing the writing of elements.

[0113] S403 . Determine in parallel whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint.

[0114] S404. Whenever information of a first checkpoint is obtained, the information of the first checkpoint is written into a circular linked list.

[0115] In some examples, after the information of the first checkpoint is written into the circular linked list, the state of the corresponding element in the circular linked list may be marked as a first state, where the first state is used to indicate that the information of the first checkpoint has been written into the element.

[0116] S405: After writing information of a preset number of first checkpoints in the circular linked list, the control thread exits.

[0117] In this embodiment, the circular linked list can only be used to accommodate a preset number of elements, that is, it can only accommodate a preset number of first checkpoint information. After the preset number of first checkpoint information is written into the circular linked list, it means that the circular linked list is full. At this time, the thread can be controlled to exit, that is, the control thread stops continuing to determine whether the log sequence number is the first checkpoint. Accordingly, the total number of log sequence numbers matched for all threads can be a preset number.

[0118] S406: Search the source cluster for a second checkpoint corresponding to the first checkpoint, and write information of the second checkpoint into a circular linked list.

[0119] For example, the checkpoint in the source cluster has a corresponding identifier, and the corresponding identifier can be determined according to the committed transaction. The checkpoint in the target cluster also has a corresponding identifier, and the corresponding identifier can also be determined according to the committed transaction. The identifier of the checkpoint in the source cluster and the identifier of the checkpoint in the target cluster can be compared to find a second checkpoint in the source cluster that corresponds to the first checkpoint.

[0120] It should be noted that when the number of first checkpoints and the number of second checkpoints are multiple, after writing the information of the first checkpoint into a certain element, if a second checkpoint corresponding to the first checkpoint is obtained, the information of the second checkpoint needs to be written into the element to ensure that the corresponding first checkpoint and second checkpoint are written into the same element.

[0121] In some examples, after the information of the second checkpoint is written into the circular linked list, the state of the corresponding element in the circular linked list may be marked as a second state, where the second state is used to indicate that the information of the second checkpoint has been written into the element.

[0122] For example, an independent thread can be started to traverse the circular linked list to see whether the information of the first checkpoint in the target cluster already exists but the information of the second checkpoint in the source cluster does not exist, obtain the hash value of the corresponding checkpoint from the source cluster, write it into the corresponding element of the circular linked list, and update the state of the element to the second state.

[0123] S407: Compare whether the information of the first checkpoint in the circular linked list is consistent with the information of the second checkpoint.

[0124] In some embodiments, only the information of the first checkpoint and the information of the second checkpoint in the element marked with the second state may be compared to see whether they are consistent, thereby reducing the number of comparisons and improving data processing efficiency.

[0125] For example, after comparing the checkpoint information in the element, the element may be marked as a third state, where the third state is used to indicate the checkpoint information in the element to complete the comparison and avoid repeated comparison.

[0126] If so, execute step S408.

[0127] S408. Obtain a second checkpoint that is consistent with the first checkpoint information.

[0128] S409: Taking the second checkpoint as the starting point, restore the data in the target cluster.

[0129] The data processing method provided in the embodiment of the present application constrains the number of log sequence numbers that need to be judged as checkpoints through a circular linked list, thereby avoiding judging too many log sequence numbers and improving data processing efficiency.

[0130] Figure 7 A structural diagram of a data processing device provided in this application, such as Figure 7 As shown, the data processing device 50 provided in this embodiment includes:

[0131] A first acquisition module 51 is used to acquire data divergence points between a source cluster and a target cluster;

[0132] A determination module 52, configured to determine in parallel whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint;

[0133] A second acquisition module 53 is used to acquire information of the first checkpoint after determining the first checkpoint;

[0134] A search module 54, configured to search the source cluster for a second checkpoint having information consistent with the first checkpoint;

[0135] The recovery module 55 is used to recover the data in the target cluster with the second checkpoint as the starting point.

[0136] In a possible implementation, the judgment module 52 is specifically used to judge in parallel through multiple threads whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint; each of the threads corresponds to at least one log sequence number before the data divergence point in the target cluster.

[0137] In a possible implementation, the data processing device also includes an allocation module, which is used to allocate a corresponding thread number to each thread in turn among the multiple threads; determine the starting log sequence number of each thread according to the data divergence point, the log block size and the thread number of each thread; determine the interval step according to the log block size and the number of threads; and allocate a corresponding log sequence number to each thread according to the starting log sequence number of each thread, the number of log sequence numbers and the interval step.

[0138] In a possible implementation, the search module 54 is specifically configured to search the source cluster for a second checkpoint that is consistent with the hash value of the first checkpoint.

[0139] In a possible implementation, the first acquisition module 51 is specifically configured to acquire the timeline of the source cluster and the timeline of the target cluster, and acquire inconsistent timelines; and determine data divergence points between the source cluster and the target cluster according to the inconsistent timelines.

[0140] In a possible implementation, the number of the first checkpoints and the number of the second checkpoints are multiple; the recovery module 55 is specifically used to find a target checkpoint that is closest in time to the data divergence point from the second checkpoint; and to recover the data in the target cluster starting from the target checkpoint.

[0141] In a possible implementation, the data processing device 50 further includes a setting module, the setting module is used to set a circular linked list, the circular linked list is used to store a preset number of elements, each of the elements includes information of a first checkpoint and / or information of a second checkpoint. Accordingly, the second acquisition module 53 is specifically used to write the information of the first checkpoint into the circular linked list each time the information of the first checkpoint is acquired; after writing the information of the preset number of first checkpoints into the circular linked list, control the thread to exit. The search module 54 is specifically used to search for a second checkpoint corresponding to the first checkpoint in the source cluster, write the information of the second checkpoint into the circular linked list; compare whether the information of the first checkpoint and the information of the second checkpoint in the circular linked list are consistent; if so, obtain a second checkpoint that is consistent with the information of the first checkpoint.

[0142] In a possible implementation, the second acquisition module 53 is further used to mark the state of the corresponding element in the circular linked list as the first state; after writing the information of the second checkpoint into the circular linked list, the state of the corresponding element in the circular linked list is marked as the second state. Correspondingly, the search module 54 is specifically used to compare whether the information of the first checkpoint in the element marked with the second state in the circular linked list is consistent with the information of the second checkpoint.

[0143] The data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0144] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.

[0145] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0146] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0147] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0148] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0149] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0150] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0151] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0152] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0153] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0154] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0158] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0159] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0160] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Get the data divergence point between the source cluster and the target cluster; Determine in parallel whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint; If yes, obtain information of the first checkpoint; Searching the source cluster for a second checkpoint that is consistent with information of the first checkpoint; Taking the second checkpoint as a starting point, data in the target cluster is restored.

2. The method according to claim 1, characterized in that The step of determining in parallel whether a plurality of log sequence numbers before the data divergence point in the target cluster correspond to a first checkpoint includes: Determine in parallel through multiple threads whether multiple log sequence numbers before the data divergence point in the target cluster correspond to the first checkpoint; Each of the threads corresponds to at least one log sequence number before the data divergence point in the target cluster.

3. The method according to claim 2, characterized in that The method further comprises: determining in parallel by multiple threads whether multiple log sequence numbers before the data divergence point in the target cluster correspond to before the first checkpoint: Assigning a corresponding thread number to each of the multiple threads in turn; Determine a starting log sequence number of each thread according to the data divergence point, the log block size, and the thread number of each thread; Determine the interval step size according to the log block size and the number of threads; According to the starting log sequence number of each thread, the number of log sequence numbers and the interval step, a corresponding log sequence number is allocated to each thread.

4. The method according to claim 1, characterized in that: The searching, in the source cluster, for a second checkpoint that is consistent with information of the first checkpoint includes: Search the source cluster for a second checkpoint that is consistent with the hash value of the first checkpoint.

5. The method according to claim 1, characterized in that The step of obtaining the data divergence point between the source cluster and the target cluster includes: Get the timeline of the source cluster and the timeline of the target cluster, and get the inconsistent timelines; A data divergence point between the source cluster and the target cluster is determined according to the inconsistent timeline.

6. The method according to any one of claims 1 to 5, characterized in that The number of the first checkpoints and the number of the second checkpoints are multiple; The recovering the data in the target cluster with the second checkpoint as the starting point includes: Searching for a target checkpoint that is closest in time to the data divergence point from the second checkpoints; Taking the target checkpoint as a starting point, data in the target cluster is restored.

7. The method according to claim 2 or 3, characterized in that: After obtaining the information of the first checkpoint, the method includes: Setting a circular linked list, the circular linked list is used to store a preset number of elements, each of the elements includes information of a first checkpoint and / or information of a second checkpoint; Each time the information of the first checkpoint is obtained, the information of the first checkpoint is written into the circular linked list; After writing the information of the preset number of first checkpoints in the circular linked list, controlling the thread to exit; Searching the source cluster for a second checkpoint that is consistent with information of the first checkpoint includes: Searching for a second checkpoint corresponding to the first checkpoint in the source cluster, and writing information of the second checkpoint into the circular linked list; Compare whether the information of the first checkpoint and the information of the second checkpoint in the circular linked list are consistent; If so, obtain a second checkpoint that is consistent with the information of the first checkpoint.

8. The method according to claim 7, characterized in that After writing the information of the first checkpoint into the circular linked list, the method further includes: Mark the state of the corresponding element in the circular linked list as the first state; After writing the information of the second checkpoint into the circular linked list, marking the state of the corresponding element in the circular linked list as the second state; The comparing whether the information of the first checkpoint and the information of the second checkpoint in the circular linked list are consistent includes: Compare whether the information of the first checkpoint in the element marked with the second state in the circular linked list is consistent with the information of the second checkpoint.

9. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images