Data verification method and device, equipment, storage medium and program product

Abstract

The invention provides a data verification method and device, equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the steps that in response to the condition that a current moment meets a preset moment, to-be-verified summary information of to-be-verified data is read from a disk space used for storing the to-be-verified data, the to-be-verified data comprises original data and redundant data obtained by encoding the original data, and the redundant data is stored in a disk space; the to-be-verified summary information comprises original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data size of the to-be-verified summary information is smaller than that of the to-be-verified data; encoding the original summary information to obtain target redundant summary information; and in response to determining that the target redundant summary information is consistent with the initial redundant summary information, determining that both the original data and the redundant data are not damaged.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of big data, and more specifically to a data verification method, device, equipment, storage medium and program product. Background Art

[0002] In a distributed file storage system, there is a certain probability that data will become inconsistent due to software bugs or silent disk errors. In this case, a Scrub (consistency check) operation is often performed regularly on the data in the cluster to detect and repair damaged data as quickly as possible to improve system reliability.

[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following problems in the related art: the data verification method in the related art needs to occupy a large amount of network bandwidth. Summary of the invention

[0004] In view of this, the present disclosure provides a data verification method, apparatus, device, storage medium and program product.

[0005] One aspect of the present disclosure provides a data verification method, comprising: in response to a current moment satisfying a preset moment, reading summary information of data to be verified from a disk space used to store the data to be verified, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data amount of the summary information to be verified is smaller than the data amount of the data to be verified; encoding the original summary information to obtain target redundant summary information; and in response to determining that the target redundant summary information is consistent with the initial redundant summary information, determining that neither the original data nor the redundant data is damaged.

[0006] According to an embodiment of the present disclosure, the data to be verified includes N data blocks stored in a distributed manner; in response to the current moment reaching a preset moment, it also includes: obtaining N storage location information for storing the N data blocks; and sending a summary information calculation instruction to a data service associated with the N storage location information, so as to sum the data information of the nth data block using finite field information in the disk space represented by the nth storage location information based on the data service response to the summary information calculation instruction, to obtain the summary information to be verified of the nth data block, wherein N is an integer greater than or equal to 2, and n is an integer determined from 1 to N in sequence.

[0007] According to an embodiment of the present disclosure, the data verification method further includes: in response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, determining that at least one data block among the N data blocks is damaged.

[0008] According to an embodiment of the present disclosure, original data is divided into K data blocks for distributed storage, and redundant data is divided into M data blocks for distributed storage, where K and M are both positive integers and K+M=N; in response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, the method further includes: randomly extracting K summary information to be verified from N summary information to be verified calculated based on the N data blocks; encoding the K summary information to be verified to obtain M redundant summary information to be verified; and in response to determining that some redundant summary information to be verified exists in the M summary information to be verified and is the same as some summary information to be verified in the M summary information to be verified except the K summary information to be verified in the N summary information to be verified, determining that the data blocks represented by the remaining summary information to be verified except the part of the summary information to be verified in the M summary information to be verified have been damaged.

[0009] According to an embodiment of the present disclosure, the data verification method also includes: obtaining first storage location information of a damaged data block and second storage location information of an undamaged data block; generating a repair instruction based on the first storage location information and the second storage location information; and sending the repair instruction to a repair module so as to repair the damaged data block based on the repair module.

[0010] According to an embodiment of the present disclosure, repairing a damaged data block based on a repair module includes: in response to a repair instruction, reading an undamaged data block from the disk space based on the second storage location information; reconstructing data based on the undamaged data block to obtain a data reconstruction result; and rewriting the data reconstruction result into the disk space represented by the first storage location information to repair the damaged data block.

[0011] According to an embodiment of the present disclosure, encoding the original summary information to obtain target redundant summary information includes: encoding the original summary information based on an erasure code algorithm to obtain target redundant summary information.

[0012] Another aspect of the present disclosure provides a data verification device, including: a summary information reading module, used to read summary information to be verified of data to be verified from a disk space used to store the data to be verified in response to a current moment satisfying a preset moment, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data amount of the summary information to be verified is smaller than the data amount of the data to be verified; an encoding module, used to encode the original summary information to obtain target redundant summary information; and a first verification module, used to determine that both the original data and the redundant data are not damaged in response to determining that the target redundant summary information is consistent with the initial redundant summary information.

[0013] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data verification method of the present disclosure.

[0014] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the data verification method of the present disclosure when executed.

[0015] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the data verification method of the present disclosure when executed.

[0016] According to the embodiments of the present disclosure, in response to the current moment satisfying the preset moment, summary information of the data to be verified is read from the disk space used to store the data to be verified, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data volume of the summary information to be verified is smaller than the data volume of the data to be verified; the original summary information is encoded to obtain target redundant summary information; and in response to determining that the target redundant summary information is consistent with the initial redundant summary information, it is determined that both the original data and the redundant data are not damaged. Since data verification can be performed based on the summary data to be verified whose data volume is smaller than that of the data to be verified, the network bandwidth used in the verification process can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 An exemplary system architecture to which the data verification method according to an embodiment of the present disclosure can be applied is schematically shown;

[0019] Figure 2 A schematic diagram schematically illustrates an algorithm principle of a data verification method according to an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of a data verification method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 4 A schematic diagram schematically shows the overall process of the data verification method according to an embodiment of the present disclosure;

[0022] Figure 5 A block diagram schematically shows a data verification device according to an embodiment of the present disclosure; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a data verification method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0029] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] In the related technologies, most of them perform consistency checks on replica data, and relatively few perform consistency checks on the entire amount of erasure coded data. Some systems will only check simple information such as the size of the data and the creation time, and are unable to perform consistency checks on the entire data. Even if some systems may perform consistency checks on the entire amount of erasure coded data, they need to read a large amount of data to the central node, which squeezes the network bandwidth of the cluster and reduces the stability of the system.

[0031] Embodiments of the present disclosure provide a data verification method, apparatus, device, storage medium and program product, the method comprising: in response to a current moment satisfying a preset moment, reading summary information of the data to be verified from a disk space used to store the data to be verified, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data amount of the summary information to be verified is less than the data amount of the data to be verified; encoding the original summary information to obtain target redundant summary information; and in response to determining that the target redundant summary information is consistent with the initial redundant summary information, determining that neither the original data nor the redundant data is damaged.

[0032] Figure 1 The exemplary system architecture 100 to which the data verification method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0037] It should be noted that the data verification method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data verification device provided in the embodiment of the present disclosure can generally be set in the server 105. The data verification method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the data verification device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the data verification method provided in the embodiment of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the data verification device provided in the embodiment of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0038] For example, the summary information to be verified may be originally stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (for example, the first terminal device 101, but not limited thereto), or stored in an external storage device and may be imported into the first terminal device 101. Then, the first terminal device 101 may locally execute the data verification method provided by the embodiment of the present disclosure, or send the summary information to be verified to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the summary information to be verified execute the data verification method provided by the embodiment of the present disclosure.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0040] Figure 2 A schematic diagram schematically illustrates an algorithm principle of a data verification method according to an embodiment of the present disclosure.

[0041] According to an embodiment of the present disclosure, when an erasure code algorithm is used, a result obtained by summing data bits and check bits within a finite field still satisfies the original EC (Erasure-Code) algorithm.

[0042] like Figure 2 As shown, taking the RS code 4 + 2 as an example, the original data bits are D1, D2, D3, D4, and then the check bits obtained after the EC algorithm are P1, P2, where the sizes of D and P are unlimited, and formula (1) can be obtained.

[0043] D1 + D2 +D3 + D4 = P1 + P2 formula (1)

[0044] After summing up the finite field of each data bit and check bit, the summary information is obtained, which are D1_d, D2_d, D3_d, D4_d, P1_d, P2_d, and formula (2) can be obtained.

[0045] D1_d + D2_d + D3_d + D4_d = P1_d + P2_d formula (2)

[0046] The same EC algorithm is applied to the summary information of the data bits to obtain the check bits of the new summary information, P1_d_n, P2_d_n, and formula (2) can be obtained.

[0047] D1_d + D2_d + D3_d + D4_d = P1_d_n + P2_d_n formula (3)

[0048] Based on formula (2) and formula (3), we can get the following formula (4) to formula (5).

[0049] P1_d = P1_d_n Formula (4)

[0050] P2_d = P2_d_n Formula (5)

[0051] Based on the above mathematical principles, the summary information of large blocks of data can be calculated on the disk and sent to the control node File Server (metadata server, providing cluster management functions). Then, the File Server uses the same EC algorithm to calculate a new check bit and compare it with the summary information read from the disk to check whether the data is damaged.

[0052] Figure 3 The flowchart of the data verification method according to the embodiment of the present disclosure is schematically shown.

[0053] like Figure 3 As shown, the method includes operations S301 to S303.

[0054] In operation S301, in response to the current moment satisfying the preset moment, summary information of the data to be verified is read from the disk space used to store the data to be verified, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, and the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data amount of the summary information to be verified is less than the data amount of the data to be verified.

[0055] According to an embodiment of the present disclosure, when new data is received in the memory space, the new data can first be segmented according to the stripe size, and the data blocks obtained by segmentation may include, for example, D1, D2, D3, and D4, which may constitute the original data. Then, the check bits may be calculated in combination with the encoding algorithm, for example, including two databases P1 and P2, which may constitute redundant data. Afterwards, D1, D2, D3, D4, P1, P2, etc. may be persisted in the disk space in units of data blocks to realize data storage. The data to be verified may include data composed of multiple data blocks such as D1, D2, D3, D4, P1, P2, etc. that have been stored in the disk space.

[0056] According to an embodiment of the present disclosure, a scheduled task may be set to execute a data verification process. The preset moment may represent the moment defined by the scheduled task. The summary information to be verified may represent the information obtained by performing a summary calculation on the data to be verified. When the summary information to be verified is calculated, the summary information to be verified may be stored together with the disk space where the relevant data to be verified is stored. The storage operation may also be omitted, but the summary information to be verified needs to be calculated for the data to be verified each time the preset moment is reached for easy reading.

[0057] According to an embodiment of the present disclosure, the original summary information is obtained by performing a summary calculation on the original data. The initial redundant summary information is obtained by performing a summary calculation on the redundant data.

[0058] According to the embodiments of the present disclosure, various digest calculation algorithms can be conceived to perform the above digest calculation process. The specific principle of the digest calculation algorithm is not limited, as long as the digest information to be verified with a smaller data volume than the data to be verified can be calculated for the data to be verified.

[0059] In operation S302, the original summary information is encoded to obtain target redundant summary information.

[0060] According to an embodiment of the present disclosure, the target redundant summary information is obtained by encoding the original summary information.

[0061] It should be noted that the encoding algorithm used in this process must be consistent with the encoding algorithm used in the aforementioned process of encoding the original data to obtain redundant data.

[0062] In operation S303, in response to determining that the target redundant summary information is consistent with the initial redundant summary information, it is determined that neither the original data nor the redundant data is damaged.

[0063] According to an embodiment of the present disclosure, based on the aforementioned principle, when it is determined that the target redundant summary information obtained by calculation is consistent with the initial redundant summary information obtained by reading from the disk space, it can be determined that the data is not damaged.

[0064] According to an embodiment of the present disclosure, the above operations S301 to S303 may be performed by the control node File Server, but are not limited thereto.

[0065] Through the above embodiments of the present disclosure, since data verification can be performed based on summary data to be verified whose data volume is smaller than that of the data to be verified, the network bandwidth used in the verification process can be reduced. Specifically, the amount of data transmitted in the network can be reduced by X times in this way, where X=the original data volume / the summary data volume.

[0066] In conjunction with specific embodiments, Figure 3 The method shown is further explained.

[0067] According to an embodiment of the present disclosure, the data to be verified may include N data blocks stored in a distributed manner. The above operation S301 may also include: obtaining N storage location information for storing the N data blocks. Sending a summary information calculation instruction to a data service associated with the N storage location information, so that based on the data service response to the summary information calculation instruction, the data information of the nth data block is summed using the finite field information in the disk space represented by the nth storage location information to obtain the summary information to be verified of the nth data block, wherein N is an integer greater than or equal to 2, and n is an integer determined from 1 to N in sequence.

[0068] According to the embodiments of the present disclosure, different databases can be distributedly stored in the same or different disk spaces. The storage location information can characterize the storage location of the data block in the disk space. Each disk space can correspond to a data service, which is responsible for persisting the data information in the data block to the disk, and can be used to support the reception of operation instructions for the data in the corresponding data block and the response to the instructions to perform computing operations on the corresponding data.

[0069] According to an embodiment of the present disclosure, the above digest calculation algorithm may include an algorithm for summing in combination with finite field information. When the algorithm is implemented, one or more finite fields may be used for calculation, as long as the finite field information used for each data block is the same, which is not limited here.

[0070] For example, the data information of the nth data block and the first finite field information may be multiplied and summed to obtain the summary information to be verified of the nth data block.

[0071] For example, the data information of the nth data block may be multiplied with the first finite field information, the second finite field information, the third finite field information, etc. in sequence and summed to obtain the summary information to be verified of the nth data block.

[0072] According to an embodiment of the present disclosure, the encoding algorithm may include an EC algorithm. The operation S302 may include: encoding the original summary information based on an erasure code algorithm to obtain target redundant summary information.

[0073] It should be noted that the above encoding algorithm is not limited to the EC algorithm, and may also include other encoding algorithms as long as it can encode the original data to obtain redundant data.

[0074] Through the above-mentioned embodiments of the present disclosure, a consistency check technology for erasure code data in a distributed storage cluster is implemented, which can detect the consistency of erasure code data in the cluster without occupying additional cluster bandwidth, thereby improving the reliability of system data.

[0075] According to an embodiment of the present disclosure, the data verification method may further include: in response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, determining that at least one data block among the N data blocks is damaged.

[0076] According to the embodiments of the present disclosure, Figure 2 According to the algorithm principle shown, it can be obtained that, when a part of the data to be verified is damaged, the summary information to be verified calculated for the damaged data block will be different from the summary information to be verified calculated for the undamaged data block, which will cause the target redundant summary information finally calculated to be different from the initial redundant summary information. Therefore, by judging the inconsistency between the target redundant summary information and the initial redundant summary information, it can be determined that one or more data blocks in the data to be verified have been damaged.

[0077] It should be noted that at this stage, it is still not possible to determine which data block is damaged. If it is necessary to determine the specific damaged data block, the following operations can be further performed.

[0078] According to an embodiment of the present disclosure, it is assumed that the original data is divided into K data blocks for distributed storage, and the redundant data is divided into M data blocks for distributed storage, K and M are both positive integers, and K+M=N. In response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, it may also include: randomly extracting K summary information to be verified from the N summary information to be verified calculated based on the N data blocks. Encode the K summary information to be verified to obtain M redundant summary information to be verified. In response to determining that there is part of the redundant summary information to be verified in the M summary information to be verified that is the same as part of the summary information to be verified in the M summary information to be verified except the K summary information to be verified in the N summary information to be verified, it is determined that the data block represented by the remaining part of the summary information to be verified in the M summary information to be verified except the part of the summary information to be verified has been damaged.

[0079] According to an embodiment of the present disclosure, when it is determined that data is damaged, K pieces can be selected from the original summary information and the initial redundant summary information, all possible permutations and combinations can be made, the remaining M summary information can be recalculated, and the settlement results can be compared to confirm whether the current block is damaged.

[0080] Combined with Figure 2 In the embodiment shown, it is assumed that K=4, M=2, and N=6. The six summary information to be verified can be represented by four original summary information, namely D1_d, D2_d, D3_d, and D4_d, and two initial redundant summary information, namely P1_d and P2_d.

[0081] For example, the target redundant summary information P1_d_n and P2_d_n can be recalculated for D1_d, D2_d, D3_d, and D4_d. If P1_d_n≠P1_d, and P2_d_n=P2_d, it can be determined that the P1 data block corresponding to P1_d is damaged. If P1_d_n≠P1_d, and P2_d_n≠P2_d, there are the following two situations: the P1 data block corresponding to P1_d and the P2 data block corresponding to P2_d are both damaged; any at least one of D1, D2, D3, and D4 is damaged. Since this situation cannot directly determine which specific data block is damaged, it can not be used as a reference for judging data block damage for this situation.

[0082] For example, the target redundant summary information D1_d_n and P2_d_n can be recalculated for D2_d, D3_d, D4_d, and P1_d. If D1_d_n≠D1_d, and P2_d_n=P2_d, it can be determined that the D1 data block corresponding to D1_d is damaged. For other situations, please refer to the above embodiments and will not be repeated here.

[0083] It should be noted that the encoding algorithm in this embodiment needs to be consistent with the encoding algorithm used in the aforementioned process of encoding the original data to obtain redundant data. This implementation can determine the damaged data blocks in the case where at most M-1 data blocks are damaged.

[0084] Through the above-mentioned embodiments of the present disclosure, it is possible to determine which specific data block is damaged, so as to facilitate the subsequent repair of the damaged data block.

[0085] According to the embodiments of the present disclosure, it can be determined whether to repair the data in combination with the upper layer business. In the case where repair is required, the first storage location information of the damaged data block and the second storage location information of the undamaged data block can be obtained. A repair instruction is generated according to the first storage location information and the second storage location information. The repair instruction is sent to the repair module so that the damaged data block is repaired based on the repair module.

[0086] According to an embodiment of the present disclosure, the above-mentioned repairing of the damaged data block based on the repair module may include: in response to the repair instruction, reading the undamaged data block from the disk space based on the second storage location information. Reconstructing the data based on the undamaged data block to obtain a data reconstruction result. Rewriting the data reconstruction result into the disk space represented by the first storage location information to repair the damaged data block.

[0087] According to an embodiment of the present disclosure, the same coding calculation can be performed on the undamaged data blocks in combination with the coding parameters consistent with the coding algorithm used in the aforementioned process of encoding the original data to obtain redundant data, so as to obtain the data reconstruction results for repairing the damaged data blocks.

[0088] According to the embodiments of the present disclosure, when there is only one damaged data block, the data reconstruction result can be directly rewritten to the disk space corresponding to the damaged data block to complete the repair. When there are multiple damaged data blocks, multiple data reconstruction results can be calculated and, according to the first storage location information, the multiple data reconstruction results can be rewritten to the corresponding first storage location randomly or based on a certain rule to achieve repair.

[0089] Figure 4 The schematic diagram schematically shows the overall process of the data verification method according to an embodiment of the present disclosure.

[0090] like Figure 4As shown, the original data D1, D2, D3, and D4 are originally read into the memory space 410, and the redundant data P1 and P2 can be calculated by the EC algorithm in the memory space 410 to obtain the data to be verified D1, D2, D3, D4, P1, and P2. The data to be verified can be stored in DataNode1, DataNode2, DataNode3, DataNode4, DataNode5, and DataNode6 respectively. DataNode corresponds to the data service and is responsible for persisting the data to the disk space 420. On this basis, the above data verification operation can be performed in conjunction with FileServer 430, and the above data repair operation can be performed in conjunction with Data Recover 440 to be responsible for repairing the damaged data blocks.

[0091] According to an embodiment of the present disclosure, File Server 430 can periodically initiate a Scrub operation to determine whether the data is sealed (data sealing, in the append write model, the sealed data will not change). If it is sealed, then send a summary calculation instruction to the DataNode cluster, calculate the summary information to be verified for each data block based on each DataNode, and obtain the original summary information calculated for the original data and the initial redundant summary information calculated for the redundant data. After each DataNode calculates the summary information to be verified, it returns it to File Server 430. File Server 430 can use the same EC algorithm for the original summary information to recalculate the target redundant summary information corresponding to the redundant data, and then compare it with the initial redundant summary information returned from the DataNode to see if they are consistent. If they are consistent, the data is not damaged, otherwise the data is damaged. And in the case of data damage, the damaged data block can be determined by the aforementioned permutation and combination method.

[0092] According to an embodiment of the present disclosure, Data Recover 440 can receive a repair instruction from File Server 430, and respond to the repair instruction, read undamaged data blocks, reconstruct the data, and rewrite the data reconstruction results into the disk space corresponding to the damaged data blocks to complete the repair.

[0093] Through the above-mentioned embodiments of the present disclosure, a Scrub algorithm for erasure coded data under the Append storage model is proposed, which can complete the data verification without occupying the network bandwidth of the cluster, thereby improving the reliability of the system.

[0094] Figure 5 A block diagram of a data verification device according to an embodiment of the present disclosure is schematically shown.

[0095] like Figure 5 As shown, the data verification device 500 includes a summary information reading module 510 , an encoding module 520 and a first verification module 530 .

[0096] The summary information reading module 510 is used to read the summary information to be verified of the data to be verified from the disk space used to store the data to be verified in response to the current moment satisfying the preset moment, wherein the data to be verified includes the original data and the redundant data obtained by encoding the original data, the summary information to be verified includes the original summary information corresponding to the original data and the initial redundant summary information corresponding to the redundant data, and the data amount of the summary information to be verified is less than the data amount of the data to be verified.

[0097] The encoding module 520 is used to encode the original summary information to obtain target redundant summary information.

[0098] The first verification module 530 is configured to determine that both the original data and the redundant data are not damaged in response to determining that the target redundant summary information is consistent with the initial redundant summary information.

[0099] According to an embodiment of the present disclosure, the data to be verified includes N data blocks stored in a distributed manner. The summary information reading module includes a storage location acquisition unit and a summary information calculation instruction sending unit.

[0100] The storage location acquisition unit is used to acquire N storage location information for storing N data blocks.

[0101] A summary information calculation instruction sending unit is used to send a summary information calculation instruction to a data service associated with N storage location information, so as to sum the data information of the nth data block by using the finite field information in the disk space represented by the nth storage location information based on the data service response to the summary information calculation instruction, so as to obtain the summary information to be verified of the nth data block, wherein N is an integer greater than or equal to 2, and n is an integer determined from 1 to N in sequence.

[0102] According to an embodiment of the present disclosure, the data verification device also includes a second verification module.

[0103] The second verification module is configured to determine that at least one data block among the N data blocks is damaged in response to determining that the target redundant summary information is inconsistent with the initial redundant summary information.

[0104] According to an embodiment of the present disclosure, the original data is divided into K data blocks for distributed storage, and the redundant data is divided into M data blocks for distributed storage, where K and M are both positive integers, and K+M=N. The second verification module further includes a summary information extraction unit, an encoding unit, and a damaged block determination unit.

[0105] The summary information extraction unit is used to randomly extract K summary information to be verified from N summary information to be verified calculated based on N data blocks.

[0106] The encoding unit is used to encode K summary information to be verified to obtain M redundant summary information to be verified.

[0107] A damaged block determining unit is used to determine that a data block represented by the remaining summary information to be verified except the part of the summary information to be verified among the M summary information to be verified is damaged in response to determining that there is some redundant summary information to be verified among the M summary information to be verified that is the same as some of the summary information to be verified among the N summary information to be verified.

[0108] According to an embodiment of the present disclosure, the data verification device further includes a storage location acquisition module, a repair instruction generation module and a repair module.

[0109] The storage location acquisition module is used to acquire first storage location information of damaged data blocks and second storage location information of undamaged data blocks.

[0110] The repair instruction generating module is used to generate a repair instruction according to the first storage location information and the second storage location information.

[0111] The repair module is used to send a repair instruction to the repair module so as to repair the damaged data block based on the repair module.

[0112] According to an embodiment of the present disclosure, the repair module includes an undamaged block reading unit, a data reconstruction unit and a rewriting unit.

[0113] The undamaged block reading unit is used to read the undamaged data block from the disk space based on the second storage location information in response to the repair instruction.

[0114] The data reconstruction unit is used to reconstruct the data based on the undamaged data blocks to obtain a data reconstruction result.

[0115] The rewriting unit is used to rewrite the data reconstruction result into the disk space represented by the first storage location information to repair the damaged data block.

[0116] According to an embodiment of the present disclosure, the encoding module includes an erasure code encoding module.

[0117] The erasure code encoding module is used to encode the original summary information based on the erasure code algorithm to obtain the target redundant summary information.

[0118] According to the embodiments of the present invention, any multiple of the modules and units, or at least part of the functions of any multiple of them, can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules and units can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules and units can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules and units can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0119] For example, any multiple of the summary information reading module 510, the encoding module 520 and the first verification module 530 can be combined in one module / unit for implementation, or any one of the modules / units can be split into multiple modules / units. Alternatively, at least part of the functions of one or more of these modules / units can be combined with at least part of the functions of other modules / units and implemented in one module / unit. According to an embodiment of the present disclosure, at least one of the summary information reading module 510, the encoding module 520 and the first verification module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the summary information reading module 510, the encoding module 520 and the first verification module 530 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0120] It should be noted that the data verification device part in the embodiment of the present disclosure corresponds to the data verification method part in the embodiment of the present disclosure. The description of the data verification device part specifically refers to the data verification method part, which will not be repeated here.

[0121] Figure 6 A block diagram of an electronic device suitable for implementing a data verification method according to an embodiment of the present disclosure is schematically shown. Figure 6The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0122] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0123] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0124] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0125] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0126] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0127] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0128] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0129] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data verification method provided by the embodiment of the present disclosure.

[0130] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0132] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0134] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data verification method, comprising: In response to the current moment satisfying the preset moment, reading summary information of the data to be verified from a disk space used to store the data to be verified, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data volume of the summary information to be verified is smaller than the data volume of the data to be verified; Encoding the original summary information to obtain target redundant summary information; and In response to determining that the target redundant summary information is consistent with the initial redundant summary information, it is determined that neither the original data nor the redundant data is damaged.

2. The method according to claim 1, wherein: The data to be verified includes N data blocks stored in a distributed manner; In response to the current time reaching the preset time, the method further includes: Acquire N storage location information for storing the N data blocks; and A summary information calculation instruction is sent to a data service associated with the N storage location information, so that based on the data service's response to the summary information calculation instruction, data information of the nth data block is summed using finite field information within the disk space represented by the nth storage location information to obtain summary information to be verified for the nth data block, wherein N is an integer greater than or equal to 2, and n is an integer determined from 1 to N in sequence.

3. The method according to claim 2, further comprising: In response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, it is determined that at least one data block among the N data blocks is damaged.

4. The method according to claim 3, wherein: The original data is divided into K data blocks for distributed storage, and the redundant data is divided into M data blocks for distributed storage, where K and M are both positive integers, and K+M=N; In response to determining that the target redundant summary information is inconsistent with the initial redundant summary information, the method further includes: Randomly extract K summary information to be verified from the N summary information to be verified calculated based on the N data blocks; Encoding the K summary information to be verified to obtain M redundant summary information to be verified; and In response to determining that part of the redundant summary information to be checked exists in the M summary information to be checked and is the same as part of the summary information to be checked in M ​​summary information to be checked except the K summary information to be checked in the N summary information to be checked, it is determined that data blocks represented by the remaining summary information to be checked in the M summary information to be checked except the part of the summary information to be checked have been damaged.

5. The method according to claim 3 or 4, further comprising: Acquire first storage location information of damaged data blocks and second storage location information of undamaged data blocks; generating a repair instruction according to the first storage location information and the second storage location information; The repair instruction is sent to a repair module so that the damaged data block is repaired based on the repair module.

6. The method according to claim 5, wherein: The repairing of the damaged data block based on the repair module includes: In response to the repair instruction, reading an undamaged data block from the disk space based on the second storage location information; Reconstructing data based on the undamaged data blocks to obtain a data reconstruction result; and The data reconstruction result is rewritten into the disk space represented by the first storage location information to repair the damaged data block.

7. The method according to claim 1, wherein: The encoding of the original summary information to obtain target redundant summary information comprises: Based on an erasure coding algorithm, the original summary information is encoded to obtain the target redundant summary information.

8. A data verification device, comprising: a summary information reading module, configured to read summary information of the data to be verified from a disk space for storing the data to be verified in response to the current moment satisfying a preset moment, wherein the data to be verified includes original data and redundant data obtained by encoding the original data, the summary information to be verified includes original summary information corresponding to the original data and initial redundant summary information corresponding to the redundant data, and the data volume of the summary information to be verified is smaller than the data volume of the data to be verified; an encoding module, used to encode the original summary information to obtain target redundant summary information; and The first verification module is configured to determine that neither the original data nor the redundant data is damaged in response to determining that the target redundant summary information is consistent with the initial redundant summary information.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images