Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention provides a data synchronization method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a first target data block in a first copy; performing data verification on the to-be-verified data in the first target data block to obtain a data verification result; if it is determined that data updating of the first target data block is not completed based on the data verification result, first metadata corresponding to the first target data block is updated based on a second target data block corresponding to the first target data block in a second copy, and then to-be-verified data in the first target data block is updated. And completing data synchronization processing of the first copy. According to the method provided by the invention, the number of io for data synchronization is reduced, the data synchronization process is simplified, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] In the field of distributed technology, a distributed block storage architecture is usually used to store data to meet the data access requirements of distributed systems. In a distributed block storage architecture, multiple copies are usually required for data redundancy. At this time, it is necessary to ensure that the data content in multiple copies is completely consistent to avoid reading different data from different copies, which may cause system errors.

[0003] Based on this, if the system or the replica fails, it is necessary to check whether the data content stored in each replica is consistent after the system is restored. If it is inconsistent, it is necessary to synchronize the data in time, and after the data synchronization is completed, provide data read and write services to the outside world.

[0004] In the related art, in order to ensure consistency among multiple copies, it is necessary to update the data of each copy consistently. However, the existing data update method has a relatively complicated process, resulting in high system delays and affecting system performance. Summary of the invention

[0005] The present application provides a data synchronization method, device, electronic device and storage medium to solve the technical problem in the prior art that after a system failure is recovered, when data is synchronized for each copy, the data update process is complicated and the latency is high, which affects the overall performance of the system.

[0006] In a first aspect, the present application provides a data synchronization method, comprising:

[0007] Determine a first target data block in the first replica; wherein the first replica represents a replica that has failed; the first target data block represents a data block that is updated when the first replica fails;

[0008] Performing data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents data to be updated when the first replica fails; and the data verification result is used to determine whether the data update of the first target data block has been completed;

[0009] If it is determined based on the data verification result that the first target data block has not completed data update, then based on the second target data block in the second copy that corresponds to the first target data block, first update the first metadata corresponding to the first target data block, and then update the data to be verified in the first target data block, to complete the data synchronization processing of the first copy; wherein, the second copy represents a copy that has not failed; and the first metadata represents attribute information of the data to be verified.

[0010] In one example, determining a first target data block in the first copy includes:

[0011] Obtaining target sequence number information corresponding to the first replica; wherein the target sequence number information is used to determine a data block in the first replica that completed data update before a failure occurred;

[0012] Based on the target sequence number information, the first target data block is determined from the data blocks included in the first copy.

[0013] In one example, the method further includes:

[0014] After completing the data synchronization process on the first copy, the target sequence number information is updated.

[0015] In one example, determining the first target data block from the data blocks included in the first copy based on the target sequence number information includes:

[0016] Determine second metadata corresponding to each data block included in the first replica; wherein the second metadata includes global sequence number information; the global sequence number information represents the data update order of each data block included in the first replica before data synchronization;

[0017] Based on the target sequence number information, target identification data is determined from the global sequence number information corresponding to each data block contained in the first copy, and the data block corresponding to the target identification data is determined as the first target data block; wherein the value of the target identification data is greater than the value of the target sequence number information.

[0018] In one example, before performing data verification on the data to be verified in the first target data block, the method further includes:

[0019] Determine data update information included in the first metadata corresponding to the first target data block; wherein the data update information includes update start position information and update length information;

[0020] Based on the update start position information and the update length information, the data to be verified is determined from the first target data block.

[0021] In one example, performing data verification on the data to be verified in the first target data block to obtain a data verification result includes:

[0022] Calculate the verification calculation data corresponding to the data to be verified according to the preset data verification method;

[0023] Compare the verification calculation data with a verification value in first metadata corresponding to the first data block to obtain a first verification result;

[0024] Obtaining first version number information included in first metadata corresponding to the first target data block, and second version number information included in third metadata corresponding to the second target data block, and comparing the first version number information with the second version number information to obtain a second verification result;

[0025] The data verification result is determined based on the first verification result and the second verification result.

[0026] In one example, determining the data verification result based on the first verification result and the second verification result includes:

[0027] If the first verification result indicates that the verification calculation data is the same as the second verification value, and the second verification result indicates that the first version number information is the same as the second version number information, then determining that the data verification result indicates that the first target data block has completed data synchronization;

[0028] If the first verification result indicates that the verification calculation data and the second verification value are different, and / or the second verification result indicates that the first version number information and the second version number information are different, then it is determined that the data verification result indicates that the first target data block has not completed data synchronization.

[0029] In a second aspect, the present application provides a data synchronization device, including:

[0030] A first determining unit is used to determine a first target data block in a first replica; wherein the first replica represents a replica that has failed; and the first target data block represents a data block that is updated when the first replica fails;

[0031] a data verification unit, configured to perform data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents data to be updated when a failure occurs in the first replica; and the data verification result is used to determine whether the data update of the first target data block has been completed;

[0032] A data synchronization unit is used to, if it is determined based on the data verification result that the first target data block has not completed data update, first update the first metadata corresponding to the first target data block based on the second target data block in the second copy corresponding to the first target data block, and then update the data to be verified in the first target data block, so as to complete the data synchronization processing of the first copy; wherein the second copy represents a copy that has not failed; and the first metadata represents attribute information of the data to be verified.

[0033] In one example, the first determining unit is configured to:

[0034] Obtaining target sequence number information corresponding to the first replica; wherein the target sequence number information is used to determine a data block in the first replica that completed data update before a failure occurred;

[0035] Based on the target sequence number information, the first target data block is determined from the data blocks included in the first copy.

[0036] In one example, the device further includes:

[0037] An updating unit is used to update the target sequence number information after completing the data synchronization processing of the first copy.

[0038] In one example, the first determining unit is configured to:

[0039] Determine second metadata corresponding to each data block included in the first replica; wherein the second metadata includes global sequence number information; the global sequence number information represents the data update order of each data block included in the first replica before data synchronization;

[0040] Based on the target sequence number information, target identification data is determined from the global sequence number information corresponding to each data block contained in the first copy, and the data block corresponding to the target identification data is determined as the first target data block; wherein the value of the target identification data is greater than the value of the target sequence number information.

[0041] In one example, the device further includes:

[0042] a second determining unit, configured to determine data update information included in first metadata corresponding to the first target data block before performing data verification on the data to be verified in the first target data block; wherein the data update information includes update start position information and update length information;

[0043] Based on the update start position information and the update length information, the data to be verified is determined from the first target data block.

[0044] In one example, the data verification unit is used to:

[0045] Calculate the verification calculation data corresponding to the data to be verified according to the preset data verification method;

[0046] Compare the verification calculation data with a verification value in first metadata corresponding to the first data block to obtain a first verification result;

[0047] Obtaining first version number information included in first metadata corresponding to the first target data block, and second version number information included in third metadata corresponding to the second target data block, and comparing the first version number information with the second version number information to obtain a second verification result;

[0048] The data verification result is determined based on the first verification result and the second verification result.

[0049] In one example, the data verification unit is used to:

[0050] If the first verification result indicates that the verification calculation data is the same as the second verification value, and the second verification result indicates that the first version number information is the same as the second version number information, then determining that the data verification result indicates that the first target data block has completed data synchronization;

[0051] If the first verification result indicates that the verification calculation data and the second verification value are different, and / or the second verification result indicates that the first version number information and the second version number information are different, then it is determined that the data verification result indicates that the first target data block has not completed data synchronization.

[0052] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0056] In a fifth aspect, the present application provides a computer program product, comprising: computer execution instructions, the computer execution instructions are stored in a readable storage medium, at least one processor of an electronic device can read the computer execution instructions from the readable storage medium, and the at least one processor executes the computer execution instructions so that the electronic device executes the method described in the first aspect.

[0057] The data synchronization method, device, electronic device and storage medium provided by the present application can obtain the first target data block in the first copy after the failure of the first copy is recovered, wherein the first target data block represents the data block that performs data update when the first copy fails. Then, the data to be verified in the first target data block can be verified to obtain the data verification result. At this time, if it is determined based on the data verification result that the first target data block has not completed data synchronization, then based on the second target data block corresponding to the first target data block in the second copy, the first metadata corresponding to the first target data block is updated first, and then the data to be verified in the first target data block is updated to complete the data synchronization processing of the first copy. In this implementation, when it is determined that the data to be verified in the first copy that has failed has not completed data update and data synchronization is required, the first metadata is updated first based on the second copy that has not failed, and then the data to be verified is updated. Compared with the existing data update method, the number of io is reduced, thereby not only reducing the disk pressure, but also reducing the delay of the io path during the data update process, improving the efficiency of data synchronization, and thus improving the overall performance of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A schematic diagram of a distributed block storage provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a process for updating data in a prior art provided in an embodiment of the present application;

[0061] Figure 3 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0062] Figure 4A schematic diagram of a data synchronization process provided in an embodiment of the present application;

[0063] Figure 5 A flowchart of another data synchronization method provided in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of global sequence number information corresponding to a data block provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of verifying calculation data provided in an embodiment of the present application;

[0066] Figure 8 A schematic diagram of a data verification result provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0068] Fig.10 A schematic diagram of the structure of another data synchronization device provided in an embodiment of the present application;

[0069] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0070] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0071] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0072] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0073] In the field of distributed technology, a distributed block storage architecture is usually used to store data to meet the data access requirements of distributed systems.

[0074] For example, Figure 1 A schematic diagram of a distributed block storage provided in an embodiment of the present application, such as Figure 1 As shown, the replica may include multiple data blocks, each of which has a metadata area and a data area. At this time, when storing data in each data block in the replica, the version number information and size information of the data to be stored can be stored in the metadata area, and the data content corresponding to the data to be stored can be stored in the data area. For example, the replica contains Figure 1 In the case of the three data blocks shown, "version number information = 1, size = 1M" can be stored in the metadata area of ​​data block 1, and the "data content" can be stored in the metadata area of ​​data block 1; "version number information = 3, size = 1M" can be stored in the metadata area of ​​data block 2, and the "data content" can be stored in the metadata area of ​​data block 2; "version number information = 1, size = 1M" can be stored in the metadata area of ​​data block 3, and the "data content" can be stored in the metadata area of ​​data block 3.

[0075] In a distributed block storage architecture, multiple copies are usually required for data redundancy. At this time, it is necessary to ensure that the data content in multiple copies is completely consistent to avoid reading different data from different copies, which may cause system errors.

[0076] For example, during the data update process, suppose that a replica / system fails. In this case, the data update operation may not be completed. In this case, it is necessary to check whether the data content stored in each replica is consistent after the system is restored. If not, data synchronization is required in a timely manner, that is, update the data of the failed replica, and provide data read and write services to the outside world after the data update is completed.

[0077] In one example, Figure 2 A schematic diagram of a flow chart of data updating in the prior art provided in an embodiment of the present application is as follows: Figure 2 As shown, in the prior art, a flag can be set in the metadata area of ​​the replica, for example, Figure 2 The modify flag shown is used to identify the data block that needs to be updated according to the modify flag.

[0078] For example, Figure 2 As shown in FIG. 1 , when the data block that needs to be updated is “data block 2”, when writing data to data block 2 in the replica, the modify flag in the metadata of data block 2 can be updated first, for example, Figure 2As shown, the modify flag can be changed to "modify = true". Then, the data is written into data block 2. After the data is written successfully, the metadata of data block 2 can be updated. For example, if the current version number information of data block 2 is 1, the version number information corresponding to data block 2 can be updated to 2, and the modify flag can be modified. That is, Figure 2 As shown, modify the modify flag to "modify = false", then the data update can be completed.

[0079] according to Figure 2 It can be seen that in the prior art, when updating the data in the replica, three IO (i.e., input and output) updates are required at the bottom layer. At the same time, the IO operation usually consumes the longest time in the storage software stack. For example, the latency of one IO operation of a mechanical disk is about 3ms, so the latency of three IO operations reaches 9ms. Therefore, the existing data update method, due to the large number of IO operations, not only increases the latency on the IO path, but also increases the storage pressure of the system and weakens the system performance.

[0080] The data synchronization method provided in this application is intended to solve the above technical problems in the prior art.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 3 A flowchart of a data synchronization method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the data synchronization method includes:

[0083] S301. Determine a first target data block in a first copy.

[0084] The first replica represents a replica that fails; the first target data block represents a data block that is updated when the first replica fails.

[0085] In one example, when the first replica fails, when updating the data blocks included in the first replica, the following situations may occur: the data blocks are not updated, the data blocks are only partially updated, the data blocks are completely updated, etc.

[0086] Since the update progress of the data update is unclear when the first replica fails, in this case, it is necessary to perform a consistency check on the data block that updated the data when the first replica failed, that is, the first target data block, after the first replica fails to recover. If the consistency check fails, the first target data block can be synchronized according to the data synchronization method provided in the embodiment of the present application. If the consistency check passes, data synchronization does not need to be performed, and data read and write services can be directly provided to the outside.

[0087] In an example, the number of the first target data blocks may be one or more.

[0088] S302: Perform data verification on the data to be verified in the first target data block to obtain a data verification result.

[0089] The data to be verified represents the data to be updated when the first replica fails; the data verification result is used to determine whether the first target data block has completed the data update.

[0090] In one example, the data to be verified can be determined based on the first metadata corresponding to the first target data block. The first metadata corresponding to the first target data block may include at least the following data: data block ID, version number information corresponding to the data update, verification value corresponding to the data update, update start position information corresponding to the data update, update length information corresponding to the data update, global sequence number information corresponding to the data update, etc. At this time, the data to be verified can be determined based on the data block ID, update start position information corresponding to the data update, and update length information corresponding to the data update.

[0091] In one example, data verification can be performed on the data to be verified in the first target data block based on the second replica that has failed, to obtain a data verification result. The data verification result may include: verification passed and verification failed. If the data verification result is verification passed, it can be determined that the first target data block has completed data update, and if the data verification result is verification failed, it can be determined that the first target data block has not completed data update.

[0092] In one example, when multiple replicas included in the distributed block storage architecture fail, the second replica can be understood as a reference replica determined from the multiple replicas, and the data to be verified in the first target data block is verified, and the first replica is synchronized based on the reference replica.

[0093] S303. If it is determined based on the data verification result that the data update of the first target data block has not been completed, based on the second target data block in the second copy corresponding to the first target data block, first update the first metadata corresponding to the first target data block, and then update the data to be verified in the first target data block to complete the data synchronization processing of the first copy.

[0094] The second copy represents a copy that has not failed; the first metadata represents attribute information of the data to be verified.

[0095] For example, Figure 4 A schematic diagram of a data synchronization process provided by an embodiment of the present application. Figure 4 As shown, when performing data synchronization, the embodiment of the present application can first update the metadata. For example, the version number information, check value, update start position information, update length information and global sequence number information in the metadata (for example, the first metadata corresponding to the first target data block) can be updated according to the data to be updated. After that, the data in the first data block (that is, the data to be checked) is updated. That is, Figure 4 The write data is shown.

[0096] It can be seen from the above description that in the embodiment of the present application, after the failure of the first replica is recovered, the first target data block in the first replica can be obtained, wherein the first target data block represents the data block for data update when the first replica fails. Then, the data to be verified in the first target data block can be verified to obtain the data verification result. At this time, if it is determined based on the data verification result that the first target data block has not completed data synchronization, then based on the second target data block corresponding to the first target data block in the second replica, the first metadata corresponding to the first target data block is first updated, and then the data to be verified in the first target data block is updated to complete the data synchronization processing of the first replica. In this implementation, when it is determined that the data to be verified in the first replica that has failed has not completed data update and data synchronization is required, the first metadata is first updated based on the second replica that has not failed, and then the data to be verified is updated. Compared with the existing data update method, the number of io is reduced, thereby not only reducing the disk pressure, but also reducing the delay of the io path during the data update process, improving the efficiency of data synchronization, and thus improving the overall performance of the storage system.

[0097] Figure 5 A flowchart of another data synchronization method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the data synchronization method includes:

[0098] S501. Obtain target sequence number information corresponding to the first copy.

[0099] The target sequence number information is used to determine the data blocks in the first replica that have completed data synchronization before the failure occurs.

[0100] In one example, the target sequence number information can be understood as the maximum sequence number identifier corresponding to the data block in the first replica that has completed data update. For example, the latest updated data block of the first replica before the failure can be determined first, and then the global sequence number information in the metadata corresponding to the latest updated data block can be determined as the maximum sequence number identifier, that is, the target sequence number information.

[0101] In one example, corresponding field information can be set for the target sequence number information, and the target sequence number information is saved to the disk at the storage location corresponding to the corresponding field information. At this time, the target sequence number information can be determined by querying the specific content corresponding to the field information in the disk.

[0102] S502: Determine a first target data block from each data block included in the first copy based on the target sequence number information.

[0103] In one example, when the first target data block is determined from the data blocks included in the first copy based on the target sequence number information, the second metadata corresponding to the data blocks included in the first copy can be determined first. Then, based on the target sequence number information, the target identification data is determined from the global sequence number information corresponding to the data blocks included in the first copy, and the data block corresponding to the target identification data is determined as the first target data block.

[0104] The second metadata includes global sequence number information; the global sequence number information represents the data update order of each data block included in the first copy before data synchronization.

[0105] Among them, the value of the target identification data is greater than the value of the target sequence number information.

[0106] In one example, a data block whose global sequence number information in the second metadata corresponding to each data block included in the first copy is greater than the target sequence information can be determined as the first target data block.

[0107] For example, Figure 6 A schematic diagram of global sequence number information corresponding to a data block provided in an embodiment of the present application. Figure 6 As shown, in the case where the data block with updated data is stored in the first copy, the global sequence number information will be updated. At this time, the global sequence number information can be obtained as follows: Figure 6As shown: global sequence number information = n, global sequence number information = n+1, global sequence number information = n+2, global sequence number information = n+3, global sequence number information = n+4. At this time, when the target sequence number information is determined to be: global sequence number information = n, then global sequence number information = n+1, global sequence number information = n+2, global sequence number information = n+3, global sequence number information = n+4 can be determined as target identification data, and the data block corresponding to the target identification data is determined as the first target data block.

[0108] At this time, the metadata of each data block in the first copy may include, in addition to the global sequence number information, the following information: Figure 6 As shown: data block ID, version number information, check value, update start position information, update length information, etc., which will not be repeated here one by one.

[0109] The above implementation can determine the first target data block for data update when the first copy fails based on the target sequence number information, so that the first target data block can be determined by judging the modify identifier of each data block contained in the first copy one by one, which simplifies the process of determining the first target data block and improves efficiency.

[0110] S503: Determine data update information included in first metadata corresponding to the first target data block.

[0111] The data update information includes update start position information and update length information.

[0112] S504: Determine the data to be verified from the first target data block based on the update start position information and the update length information.

[0113] For example, when the first target data block is Figure 4 In the case of "data block 2" shown, the update start position is "0" and the update length information is "4096". At this time, the data in the first target data block, starting from the update start position information and with a length equal to the update length information, can be determined as the data to be verified.

[0114] This implementation manner can determine the relevant information of the updated data in each data block in the first copy as the attribute information of the data block, that is, the first metadata, so that the data to be verified can be determined simply and quickly based on the first metadata.

[0115] S505: Perform data verification on the data to be verified in the first target data block to obtain a data verification result.

[0116] The data to be verified represents the data to be updated when the first replica fails; the data verification result is used to determine whether the first target data block has completed the data update.

[0117] In one example, data verification is performed on the data to be verified in the first target data block to obtain a data verification result, which can be described as the following process.

[0118] First, according to a preset data verification method, the verification calculation data corresponding to the data to be verified is calculated.

[0119] Among them, the pre-set data verification method can be a pre-set cyclic redundancy check method. For example, the pre-set data verification method can be CRC64. The pre-set data verification method is not specifically limited here, and is subject to achievable implementation.

[0120] In one example, Figure 7 A schematic diagram of a verification calculation data provided in an embodiment of the present application. Figure 7 As shown, assuming that before the data to be verified is updated, the verification value of the data to be verified is "verification value = 0x12345678", then, in the case where the data to be verified is not written (that is, Figure 7 In the case of case 1 shown in the figure, the calculated verification calculation data of the data to be verified is "verification value = 0x12345678"; in the case of partial writing of the data to be verified (that is, Figure 7 In the case of the second case shown in the figure, the calculated verification data of the data to be verified is "verification value = 0x87659076"; when the data to be verified is written (that is, Figure 7 As shown in case 3), the verification calculation data of the data to be verified is calculated as "verification value = 0x23345409".

[0121] Then, the verification calculation data is compared with the verification value in the first metadata corresponding to the first data block to obtain a first verification result.

[0122] In one example, the first verification result may include: verification pass information and verification fail information, wherein the verification pass information indicates that the verification calculation data and the second verification value are the same, and the verification fail information indicates that the verification calculation data and the second verification value are different.

[0123] Afterwards, the first version number information included in the first metadata corresponding to the first target data block and the second version number information included in the third metadata corresponding to the second target data block are obtained, and the first version number information and the second version number information are compared to obtain a second verification result.

[0124] The second verification result may include: verification pass information and verification fail information. The verification pass information indicates that the first version number information and the second version number information are the same, and the verification fail information indicates that the first version number information and the second version number information are different.

[0125] Finally, based on the first verification result and the second verification result, a data verification result is determined.

[0126] In one example, if the first verification result indicates that the verification calculation data is the same as the second verification value, and the second verification result indicates that the first version number information is the same as the second version number information, then it is determined that the data verification result indicates that the first target data block has completed data synchronization.

[0127] If the first verification result indicates that the verification calculation data is different from the second verification value, and / or the second verification result indicates that the first version number information is different from the second version number information, then it is determined that the data verification result indicates that the first target data block has not completed data synchronization.

[0128] In one example, Figure 8 A schematic diagram of a data verification result provided in an embodiment of the present application. Figure 8 As shown, for the data block with data block ID=2, the version number information of the first copy and the second copy are the same (that is, the second verification result is the verification pass information), but the verification value on the first copy fails (that is, the first verification result is the verification fail information). At this time, it is necessary to synchronize the data block with data block ID=2 in the first copy and the data block with ID=2 on the second copy to restore consistency; for the data block with data block ID=10, if the version number information and the verification value of the two copies are verified, data synchronization is not required; for the data block with data block ID=3, although the verification value of the first copy and the second copy is verified, the version number information is different, and data synchronization is also required.

[0129] In the above implementation, the verification data can be treated as data based on the metadata of the data block, and the data verification result can be obtained, so that the inconsistent data between the replicas can be detected based on the partial writing of the local data, which can improve the efficiency of consistency detection between the replicas and further improve the efficiency of data synchronization.

[0130] S506. If it is determined based on the data verification result that the data update of the first target data block has not been completed, based on the second target data block in the second copy corresponding to the first target data block, first update the first metadata corresponding to the first target data block, and then update the data to be verified in the first target data block to complete the data synchronization processing of the first copy.

[0131] The second copy represents a copy that has not failed; the first metadata represents attribute information of the data to be verified.

[0132] In an example, this step can refer to the step described in the above S303, and will not be repeated here in detail.

[0133] S507: After completing the data synchronization process on the first copy, update the target sequence number information.

[0134] In one example, after completing the data synchronization processing of the first copy, the target sequence number information can be updated according to the global sequence number information corresponding to the last data block that completed the data synchronization processing. At this time, the updated target sequence number information can be used to overwrite the original target sequence number information, thereby achieving timely update of the target sequence number information and facilitating the next data synchronization.

[0135] Fig. 9 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the data synchronization device 900 includes:

[0136] The first determination unit 901 is used to determine a first target data block in a first replica, wherein the first replica represents a replica that has failed; and the first target data block represents a data block that is updated when the first replica fails.

[0137] The data verification unit 902 is used to perform data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents the data to be updated when a failure occurs in the first copy; the data verification result is used to determine whether the first target data block has completed the data update.

[0138] The data synchronization unit 903 is used to, if it is determined based on the data verification result that the data update of the first target data block has not been completed, first update the first metadata corresponding to the first target data block based on the second target data block corresponding to the first target data block in the second copy, and then update the data to be verified in the first target data block, so as to complete the data synchronization processing of the first copy; wherein the second copy represents a copy that has not failed; and the first metadata represents the attribute information of the data to be verified.

[0139] Fig.10 A structural diagram of another data synchronization device provided in an embodiment of the present application, such as Fig.10 As shown, the data synchronization device 1000 includes:

[0140] The first determination unit 1001 is used to determine a first target data block in a first replica, wherein the first replica represents a replica that has failed; and the first target data block represents a data block that is updated when the first replica fails.

[0141] The data verification unit 1002 is used to perform data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents the data to be updated when a failure occurs in the first copy; the data verification result is used to determine whether the first target data block has completed the data update.

[0142] The data synchronization unit 1003 is used to, if it is determined based on the data verification result that the first target data block has not completed data update, first update the first metadata corresponding to the first target data block based on the second target data block in the second copy corresponding to the first target data block, and then update the data to be verified in the first target data block, so as to complete the data synchronization processing of the first copy; wherein the second copy represents a copy that has not failed; and the first metadata represents the attribute information of the data to be verified.

[0143] In one example, the first determining unit 1001 is configured to:

[0144] Obtain target sequence number information corresponding to the first replica; wherein the target sequence number information is used to determine a data block in the first replica that completed data update before the failure occurred;

[0145] Based on the target sequence number information, a first target data block is determined from the data blocks included in the first copy.

[0146] In one example, the device further includes:

[0147] The updating unit 1004 is used to update the target sequence number information after completing the data synchronization process on the first copy.

[0148] In one example, the first determining unit 1001 is configured to:

[0149] Determine the second metadata corresponding to each data block included in the first replica; wherein the second metadata includes global sequence number information; the global sequence number information represents the data update order of each data block included in the first replica before data synchronization;

[0150] Based on the target sequence number information, the target identification data is determined from the global sequence number information corresponding to each data block contained in the first copy, and the data block corresponding to the target identification data is determined as the first target data block; wherein the value of the target identification data is greater than the value of the target sequence number information.

[0151] In one example, the device further includes:

[0152] The second determining unit 1005 is used to determine the data update information included in the first metadata corresponding to the first target data block before performing data verification on the data to be verified in the first target data block; wherein the data update information includes update start position information and update length information;

[0153] Based on the update start position information and the update length information, the data to be verified is determined from the first target data block.

[0154] In one example, the data verification unit 1002 is used to:

[0155] According to the preset data verification method, the verification calculation data corresponding to the data to be verified is calculated;

[0156] Determine a second target data block in the second copy that corresponds to the first target data block, and obtain verification reference data in the second target data block that corresponds to the data to be verified;

[0157] Determine a second verification value corresponding to the verification reference data, and compare the verification calculation data with the second verification value to obtain a first verification result;

[0158] Obtaining first version number information included in first metadata corresponding to the first target data block and second version number information included in third metadata corresponding to the second target data block, and comparing the first version number information with the second version number information to obtain a second verification result;

[0159] Based on the first verification result and the second verification result, a data verification result is determined.

[0160] In one example, the data verification unit 1002 is used to:

[0161] If the first verification result indicates that the verification calculation data and the second verification value are the same, and the second verification result indicates that the first version number information and the second version number information are the same, then determining that the data verification result indicates that the first target data block has completed data synchronization;

[0162] If the first verification result indicates that the verification calculation data is different from the second verification value, and / or the second verification result indicates that the first version number information is different from the second version number information, then it is determined that the data verification result indicates that the first target data block has not completed data synchronization.

[0163] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the electronic device 1100 includes: a memory 1101 and a processor 1102 .

[0164] Memory 1101 is a memory used to store instructions executable by processor 1102 .

[0165] The processor 1102 is configured to execute the method provided in the above embodiment.

[0166] The electronic device further includes a receiver 1103 and a transmitter 1104. The receiver 1103 is used to receive instructions and data sent by an external device, and the transmitter 1104 is used to send instructions and data to the external device.

[0167] The present application also provides a computer-readable storage medium on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the steps of the data synchronization method in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0168] An embodiment of the present application also provides a computer program product, which carries computer execution instructions. The instructions included in the computer execution instructions can be used to execute the steps of the data synchronization method in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0169] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0170] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0171] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0172] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0173] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0174] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for an electronic device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0177] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Determine a first target data block in the first replica; wherein the first replica represents a replica that has failed; the first target data block represents a data block that is updated when the first replica fails; Performing data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents data to be updated when the first replica fails; and the data verification result is used to determine whether the data update of the first target data block has been completed; If it is determined based on the data verification result that the first target data block has not completed data update, then based on the second target data block in the second copy that corresponds to the first target data block, first update the first metadata corresponding to the first target data block, and then update the data to be verified in the first target data block, to complete the data synchronization processing of the first copy; wherein, the second copy represents a copy that has not failed; and the first metadata represents attribute information of the data to be verified.

2. The method according to claim 1, characterized in that The determining of the first target data block in the first copy includes: Obtaining target sequence number information corresponding to the first replica; wherein the target sequence number information is used to determine a data block in the first replica that completed data update before a failure occurred; Based on the target sequence number information, the first target data block is determined from the data blocks included in the first copy.

3. The method according to claim 2, characterized in that The method further comprises: After completing the data synchronization process on the first copy, the target sequence number information is updated.

4. The method according to claim 2, characterized in that: Determining the first target data block from the data blocks included in the first copy based on the target sequence number information includes: Determine second metadata corresponding to each data block included in the first replica; wherein the second metadata includes global sequence number information; the global sequence number information represents the data update order of each data block included in the first replica before data synchronization; Based on the target sequence number information, target identification data is determined from the global sequence number information corresponding to each data block contained in the first copy, and the data block corresponding to the target identification data is determined as the first target data block; wherein the value of the target identification data is greater than the value of the target sequence number information.

5. The method according to claim 1, characterized in that Before performing data verification on the data to be verified in the first target data block, the method further includes: Determine data update information included in the first metadata corresponding to the first target data block; wherein the data update information includes update start position information and update length information; Based on the update start position information and the update length information, the data to be verified is determined from the first target data block.

6. The method according to claim 1, characterized in that Performing data verification on the data to be verified in the first target data block to obtain a data verification result includes: Calculate the verification calculation data corresponding to the data to be verified according to the preset data verification method; Compare the verification calculation data with a verification value in first metadata corresponding to the first data block to obtain a first verification result; Obtaining first version number information included in first metadata corresponding to the first target data block, and second version number information included in third metadata corresponding to the second target data block, and comparing the first version number information with the second version number information to obtain a second verification result; The data verification result is determined based on the first verification result and the second verification result.

7. The method according to claim 6, characterized in that The determining the data verification result based on the first verification result and the second verification result includes: If the first verification result indicates that the verification calculation data is the same as the second verification value, and the second verification result indicates that the first version number information is the same as the second version number information, then determining that the data verification result indicates that the first target data block has completed data synchronization; If the first verification result indicates that the verification calculation data and the second verification value are different, and / or the second verification result indicates that the first version number information and the second version number information are different, then it is determined that the data verification result indicates that the first target data block has not completed data synchronization.

8. A data synchronization device, characterized in that: include: A first determining unit is used to determine a first target data block in a first replica; wherein the first replica represents a replica that has failed; and the first target data block represents a data block that is updated when the first replica fails; a data verification unit, configured to perform data verification on the data to be verified in the first target data block to obtain a data verification result; wherein the data to be verified represents data to be updated when a failure occurs in the first replica; and the data verification result is used to determine whether the data update of the first target data block has been completed; A data synchronization unit is used to, if it is determined based on the data verification result that the first target data block has not completed data update, first update the first metadata corresponding to the first target data block based on the second target data block in the second copy corresponding to the first target data block, and then update the data to be verified in the first target data block, so as to complete the data synchronization processing of the first copy; wherein the second copy represents a copy that has not failed; and the first metadata represents attribute information of the data to be verified.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data synchronization method according to any one of claims 1 to 7.

Images