Data verification method, distributed block storage system and data restoration method

Abstract

The invention relates to a data verification method, a distributed block storage system and a data recovery method. The method comprises the following steps: sending a data verification request to slave storage equipment of to-be-verified fragmented data according to a master copy; wherein a slave copy of the to-be-verified fragmented data is stored in the slave storage device; receiving a data verification response sent by the slave storage device; wherein the data verification response is generated by the slave storage device according to the slave copy and the data verification request; and determining a data verification result of the to-be-verified fragmented data according to the primary copy and the data verification response. By adopting the method, the reliability of the distributed storage system can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data verification, and in particular to a data verification method, a distributed block storage system, and a data repair method. Background Art

[0002] Distributed block storage system divides data according to certain rules and stores it in multiple independent general storage servers. Distributed block storage system adopts a scalable system structure, uses multiple storage servers to share the storage load, and uses location servers to locate storage information. It has high availability and access efficiency, and is easy to expand.

[0003] However, the reliability of the distributed block storage system in the related art is low. Summary of the invention

[0004] Based on this, it is necessary to provide a data verification method, a distributed block storage system and a data repair method that can improve the reliability of a distributed storage system in response to the above technical problems.

[0005] In a first aspect, the present application provides a data verification method, applied to a primary storage device, the method comprising:

[0006] Sending a data verification request to a slave storage device of the shard data to be verified according to the master copy of the shard data to be verified; wherein the slave storage device stores a slave copy of the shard data to be verified;

[0007] Receiving a data verification response sent by the slave storage device; wherein the data verification response is generated by the slave storage device according to the slave copy and the data verification request;

[0008] The data verification result of the shard data to be verified is determined according to the primary copy and the data verification response.

[0009] In one embodiment, the shard data to be verified includes a master copy and at least two slave copies, the master copy and at least two slave copies of the shard data to be verified are stored in different storage devices, respectively, the data verification response includes the slave copy stored in the slave storage device, and determining the data verification result of the shard data to be verified according to the master copy and each of the data verification responses includes:

[0010] When the slave copies are the same and the master copy is different from the slave copy, determining that the data verification result of the shard data to be verified is that the master copy is abnormal;

[0011] When the target slave replica among at least two of the slave replicas is different from the remaining slave replicas and the master replica, and the master replica is the same as the remaining slave replicas, it is determined that the data verification result of the shard data to be verified is that the target slave replica is abnormal.

[0012] In one embodiment, the method further comprises:

[0013] When the master copy is abnormal, calibrating the data of the master copy according to the slave copy;

[0014] In the case that the target slave replica is abnormal, data calibration is performed on the target slave replica according to the master replica.

[0015] In one embodiment, the primary storage device includes metadata and the shard data to be verified, and sending a data verification request to a slave storage device of the shard data to be verified based on the primary copy of the shard data to be verified includes:

[0016] Determine a slave storage device of the shard data to be verified according to the metadata and the master copy of the shard data to be verified; wherein the metadata includes a mapping relationship between the copy of the shard data to be verified and the storage device;

[0017] A data verification request is generated according to the master copy and sent to the slave storage device.

[0018] In one embodiment, the primary storage device includes a plurality of data blocks; among the plurality of data blocks, a metadata data block is used to store the metadata, and a target data block is used to store a primary copy of the shard data to be verified; and generating a data verification request according to the primary copy and sending it to the slave storage device includes:

[0019] Reading multiple data clusters of the target data block to obtain a master copy of the shard data to be verified;

[0020] Encapsulating the obtained master copy and generating the data verification request;

[0021] A data verification request is sent to the slave storage device.

[0022] In one embodiment, the method further comprises:

[0023] receiving a data recovery request sent by a control device;

[0024] Determine the shard data to be restored and the target storage device according to the data recovery request;

[0025] A write request is sent to the target storage device according to the primary copy of the shard data to be restored, so as to write the secondary copy of the shard data to be restored in the target storage device.

[0026] In a second aspect, the present application provides a data verification method, applied to a slave storage device, the method comprising:

[0027] Receive a data verification request sent by a primary storage device; wherein the data verification request includes a primary copy of the shard data to be verified;

[0028] Determine whether a slave copy of the shard data to be verified is stored according to the master copy of the shard data to be verified;

[0029] When it is determined that the slave copy is stored, a data verification response is generated based on the slave copy of the shard data to be verified and sent to the primary storage device, so that the primary storage device determines the data verification result of the shard data to be verified based on the data verification response.

[0030] In one of the embodiments, determining whether a slave copy of the shard data to be verified is stored according to the master copy of the shard data to be verified includes:

[0031] Get the version number of each shard data stored;

[0032] Searching for the version number of each stored shard data according to the version number of the shard data to be verified corresponding to the primary copy;

[0033] In the case of having the same version number, determine the slave copy storing the shard data to be verified.

[0034] In a third aspect, the present application provides a distributed block storage system, the system comprising multiple storage devices, wherein a master storage device among the multiple storage devices is used to store a master copy of the shard data to be verified, and is used to verify the shard data using any of the data verification methods described in the above embodiments applied to the master storage device, and a slave storage device among the multiple storage devices is used to store a slave copy of the shard data to be verified, and executes any of the data verification methods described in the above embodiments applied to the slave storage device.

[0035] In a fourth aspect, the present application provides a data repair method, which is applied to a control device in the distributed block storage system described in the above embodiment, wherein the control device is respectively connected to multiple storage devices in the distributed block storage system; the method comprises:

[0036] Obtaining current status information of each of the storage devices;

[0037] Determining a faulty storage device among the storage devices according to current status information of the storage devices;

[0038] According to normal storage devices among the plurality of storage devices, target storage devices corresponding to each slice data in the faulty storage device are respectively determined, so that each slice data in the faulty storage device is correspondingly written into each target storage device.

[0039] In one embodiment, the determining, based on normal storage devices among the plurality of storage devices, target storage devices corresponding to each shard data in the faulty storage device, so as to write each shard data in the faulty storage device into each target storage device, respectively, includes:

[0040] According to normal storage devices among the plurality of storage devices, respectively determine the copy type and target storage device corresponding to each shard data in the faulty storage device;

[0041] For any shard data in the failed storage device, if the copy type of the shard data is a slave copy, send a data recovery request to a normal storage device storing a master copy of the shard data, so as to write the slave copy of the shard data to the target storage device;

[0042] In the case where the copy type of the shard data is a master copy, any normal storage device storing a slave copy of the shard data is determined as the master storage device of the shard data, and a data recovery request is sent to the master storage device of the shard data to write the slave copy of the shard data to the target storage device.

[0043] The above-mentioned data verification method, distributed block storage system and data repair method include: sending a data verification request to a slave storage device of the shard data to be verified according to the master copy, wherein the slave storage device stores the slave copy of the shard data to be verified; receiving a data verification response sent by the slave storage device, wherein the data verification response is generated from the storage device according to the slave copy and the data verification request; determining the data verification result of the shard data to be verified according to the master copy and the data verification response. The data verification method of the present application can send a data verification request to the storage device storing the corresponding slave copy according to the data content of the master copy, so that the storage device storing the corresponding slave copy takes out the corresponding data content from the slave copy according to the data content of the master copy, and sends the data verification response to the master storage device storing the master copy. After receiving the data verification response, the master storage device can perform data verification according to the data content of the master copy and the data content corresponding to the slave copy in the data verification response, and judge whether there is a data content inconsistency between the master copy and the slave copy. If so, the abnormal data content can be repaired in time, thereby improving the reliability of the data content stored in each storage device, that is, improving the reliability of the distributed block storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] FIG1 is a schematic diagram of a flow chart of a data verification method applied to a primary storage device in one embodiment;

[0046] FIG2 is a schematic diagram of the structure of a distributed block storage system in one embodiment;

[0047] FIG3 is a schematic diagram of a flow chart of step S103 in one embodiment;

[0048] FIG4 is a schematic diagram of a flow chart of step S101 in one embodiment;

[0049] FIG5 is a schematic diagram of a flow chart of step S402 in one embodiment;

[0050] FIG6 is a schematic flow chart of a data verification method applied to a primary storage device in another embodiment;

[0051] FIG7 is a schematic diagram of a flow chart of a data verification method applied to a slave storage device in one embodiment;

[0052] FIG8 is a schematic diagram of a flow chart of step S702 in one embodiment;

[0053] FIG9 is a schematic diagram of a flow chart of a data repair method in one embodiment;

[0054] FIG10 is a schematic diagram of a flow chart of step S903 in one embodiment;

[0055] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In an exemplary embodiment, referring to FIG. 1 , the present application provides a data verification method, which is applied to a primary storage device, and the method includes steps S101 to S103 .

[0058] S101: Sending a data verification request to a slave storage device of the shard data to be verified according to the master copy of the shard data to be verified, wherein the slave storage device stores a slave copy of the shard data to be verified.

[0059] Distributed block storage system is a data storage technology that utilizes the disk space on each machine in the enterprise through the network, and forms these scattered storage resources into a virtual storage device to achieve distributed data storage. Distributed storage block system has the advantages of scalability, high availability, low cost, elastic storage and high performance. Specifically, the distributed storage system can be expanded to hundreds or even thousands of nodes, and the overall performance of the system can grow linearly with the increase in the number of nodes; data is stored in multiple nodes in a distributed manner, and even if some nodes fail, the system can still continue to operate, and redundant backup can be used to ensure that data will not be lost; distributed storage systems can use cheap hardware devices, such as ordinary PC servers, and reduce operation and maintenance costs through automatic fault tolerance and automatic load balancing technology; data storage can be flexibly increased or reduced according to business needs without interrupting system operation; distributed storage systems usually have higher performance because they can process data in parallel on multiple nodes and use advanced technologies such as data sharding, data caching, and load balancing.

[0060] In the application, the distributed block storage system may have multiple storage devices BSD, each storage device may have at least one metadata data block and multiple common data blocks Blob, the common data block Blob is used to store Shard data, the metadata data block is used to store metadata, metadata refers to the parameter information of the distributed block storage system and the storage device, such as the unique identifier of the corresponding storage device BSD, the status of the corresponding storage device BSD (normal operation, stopped operation, removed from the cluster, etc.), the weight of the corresponding storage device BSD (the weight is used to determine the relative importance of BSD in data distribution and replication), the location information of the corresponding storage device BSD (host name and device path, used to identify the location of BSD), the mapping relationship between shard data and storage devices (the mapping relationship between the copy of each shard data Shard and BSD), and the peer relationship between storage devices (BSDs storing the same shard data Shard are used for data replication and recovery process). Users can use the block storage service by creating, modifying, and deleting storage volumes. In high-performance storage scenarios, due to the use of user-mode network and storage technologies, the storage volumes are sharded into larger sizes to reduce the amount of distributed metadata and management costs. For example, the size of each shard data can be 64GB.

[0061] In one example, please refer to FIG. 2, the storage volume Volume has two shard data Shard1 and Shard2, and the distributed storage system has four storage devices BSD1~BSD4, and each storage device has three common data blocks Blob1~Blob3. Each shard data Shard can have a master copy and multiple slave copies, and each copy is stored on a different storage device. For example, the master copy of the shard data Shard1 is stored in Blob1 of the storage device BSD1, and the two slave copies of the shard data Shard1 are stored in Blob2 of the storage device BSD2 and Blob3 of the storage device BSD4, respectively. The master copy of the shard data Shard2 is stored in Blob3 of the storage device BSD2, and the two slave copies of the shard data Shard2 are stored in Blob3 of the storage device BSD3 and Blob2 of the storage device BSD4, respectively. The storage device storing the master copy of the shard data is the master storage device of the shard data, and the storage device storing the slave copy of the shard data is the slave storage device of the shard data. In this example, the master storage device corresponding to the shard data Shard1 is storage device BSD1, and the two slave storage devices are storage device BSD2 and storage device BSD4; the master storage device corresponding to the shard data Shard2 is storage device BSD2, and the two slave storage devices are storage device BSD3 and storage device BSD4. For the shard data Shard1, storage device BSD1, storage device BSD2, and storage device BSD4 are storage devices BSD that have a peer relationship with each other; for the shard data Shard2, storage device BSD2, storage device BSD3, and storage device BSD4 are storage devices BSD that have a peer relationship with each other.

[0062] In the actual application of the distributed block storage system, since the storage service is built on physical devices, that is, each storage device BSD is a physical device, and the physical device may have silent errors that cause errors in the stored data, and the abnormality of the software system may also cause errors in the data stored in the storage system. Therefore, the present application can set the master storage device of the shard data to periodically take out at least part of the data content of the master copy of the shard data, encapsulate it, and send a data verification request to the corresponding slave storage device of the shard data through the network module of the storage device to start data verification.

[0063] S102: Receive a data verification response sent from a storage device, wherein the data verification response is generated from the storage device according to the replica and the data verification request.

[0064] After receiving the data verification request sent by the primary storage device, the slave storage device can search whether there is a slave copy of the sharded data stored, and retrieve the corresponding data content from the slave copy based on the partial data content of the primary copy in the data verification request, encapsulate the data to form a data verification response, and feed back the data verification response to the primary storage device.

[0065] S103: Determine the data verification result of the shard data to be verified according to the primary copy and the data verification response.

[0066] After the master storage device receives the data verification responses sent by each slave storage device, it can parse each data verification response, read the data content corresponding to the slave copy, and compare the data content of the master copy and each slave copy to determine whether there is any abnormality in the data stored in the master copy and each slave copy. If so, perform data repair in a timely manner.

[0067] The above-mentioned data verification method includes: sending a data verification request to a slave storage device of the shard data to be verified according to the master copy, wherein the slave storage device stores the slave copy of the shard data to be verified; receiving a data verification response sent by the slave storage device, wherein the data verification response is generated from the storage device according to the slave copy and the data verification request; determining the data verification result of the shard data to be verified according to the master copy and the data verification response. The data verification method of the present application can send a data verification request to the storage device storing the corresponding slave copy according to the data content of the master copy, so that the storage device storing the corresponding slave copy takes out the corresponding data content from the slave copy according to the data content of the master copy, and sends the data verification response to the master storage device storing the master copy. After receiving the data verification response, the master storage device can perform data verification according to the data content of the master copy and the data content corresponding to the slave copy in the data verification response, and judge whether there is a data content inconsistency between the master copy and the slave copy. If so, the abnormal data content can be repaired in time, thereby improving the reliability of the data content stored in each storage device, that is, improving the reliability of the distributed block storage system.

[0068] In an exemplary embodiment, please refer to FIG. 3 , step S103 , determining the data verification result of the shard data to be verified according to the master copy and each data verification response, including step S301 and step S302 .

[0069] S301: When the slave copies are the same and the master copy is different from the slave copy, it is determined that the data verification result of the shard data to be verified is abnormal for the master copy.

[0070] If the data contents of the slave copies fed back by each slave storage device are the same but different from the data contents of the master copy, the master storage device BSD determines that the data contents of this part of the stored master copy are incorrect and performs data calibration on the master copy based on the data contents of the slave copies.

[0071] S302: When the target slave replica among at least two slave replicas is different from the remaining slave replicas and the master replica, and the master replica is the same as the remaining slave replicas, it is determined that the data verification result of the shard data to be verified is that the target slave replica is abnormal.

[0072] If there is a slave replica whose data content is inconsistent with the data content of the master replica and the data content of the other slave replicas, it indicates that there is an abnormality in the slave replica. The slave replica is marked as the target slave replica, and the master storage device can send a data repair request to the slave storage device where the target slave replica is located. When the slave storage device where the target slave replica is located receives the data repair request, it can calibrate the data of the target slave replica according to the data content of the master replica in the data verification request previously received.

[0073] In an exemplary embodiment, please refer to FIG. 4 , step S101 , sending a data verification request to a slave storage device of the shard data to be verified based on the primary copy of the shard data to be verified, including step S401 and step S402 .

[0074] S401: Determine a slave storage device for the shard data to be verified according to the metadata and the master copy of the shard data to be verified.

[0075] Before starting data verification, the main storage device first determines which storage devices the slave copies corresponding to the master copy in the main storage device are located in based on the stored metadata, which includes the mapping relationship between the copies of the sharded data to be verified and the storage devices, that is, obtains the storage devices corresponding to the main storage device.

[0076] S402: Generate a data verification request based on the master copy and send it to the slave storage device.

[0077] After determining each slave storage device, the master storage device can read the data content of the master copy of the shard data to be verified, encapsulate the data content of the master copy and generate a data verification request, and then send the data verification request to each slave storage device respectively.

[0078] In an exemplary embodiment, the primary storage device includes multiple data blocks, which can be divided into metadata data blocks and ordinary data blocks. The metadata data blocks are used to store metadata, and the ordinary data blocks are used to save copies of shard data. The target data blocks in the ordinary data blocks are used to store the master copy of the shard data to be verified. Please refer to Figure 5, step S402, generating a data verification request according to the master copy and sending it to the slave storage device, including step S501 and step S503.

[0079] S501: Read multiple data clusters of the target data block to obtain the master copy of the shard data to be verified.

[0080] It can be understood that a data block Blob can be composed of M data clusters Cluster, and the copy of the sharded data will occupy N data clusters Cluster in the data block Blob. The primary storage device can read the data content in K data clusters in the target data block to obtain at least part of the data content of the primary copy of the sharded data to be verified, 0<K≤N≤M.

[0081] S502: Encapsulate the acquired master copy and generate a data verification request.

[0082] After obtaining the data contents in the K data clusters in the target data block, the primary storage device may encapsulate the data contents in the K data clusters and generate a data verification request.

[0083] S503: Send a data verification request to the secondary storage device.

[0084] Afterwards, the master storage device can send a data verification request to each slave storage device to start data verification. In the application, after the data content in the K data clusters of the master copy is verified, the master storage device can check whether there are any unread data clusters in the target data block. If so, read the data content in the remaining data clusters and repeat the data verification process until all the data content in the target data block is verified.

[0085] In an exemplary embodiment, referring to FIG. 6 , the data verification method further includes steps S601 to S603 .

[0086] S601: Receive a data recovery request sent by a control device.

[0087] In the application, the distributed block storage system may also include a control device, which can monitor the operating status of each storage device. If a faulty or offline storage device is found, the control device can determine which copies of shard data are stored in the faulty storage device based on the metadata stored in the remaining normal storage devices, the metadata including the mapping relationship between shard data and storage devices.

[0088] S602: Determine the shard data to be restored and the target storage device according to the data recovery request.

[0089] If the failed storage device stores a slave copy of the shard data to be recovered, the controller sends a data recovery request to the primary storage device, and the primary storage device determines the shard data to be recovered based on the data recovery request, and determines the target storage device from the remaining normal storage devices.

[0090] S603: Send a write request to the target storage device according to the primary copy of the shard data to be restored, so as to write the secondary copy of the shard data to be restored in the target storage device.

[0091] After determining the target storage device from the remaining normal storage devices, the primary storage device can read all the data contents of the primary copy of the shard data to be recovered, encapsulate the data to generate a write request, and send it to the target storage device, so that the target storage device writes the data content of the shard data to be recovered in the idle ordinary data block according to the primary copy, and generates a new slave copy of the shard data to be recovered.

[0092] In an exemplary embodiment, referring to FIG. 7 , the present application provides a data verification method, which is applied to a storage device, and includes steps S701 to S703 .

[0093] S701: Receive a data verification request sent by a primary storage device, wherein the data verification request includes a primary copy of the shard data to be verified.

[0094] First, a data verification request sent by a primary storage device is received from a storage device, where the data verification request includes not only at least part of the data content of the primary copy of the shard data to be verified, but also the version number of the shard data to be verified.

[0095] S702: Determine whether a slave copy of the shard data to be verified is stored according to the master copy of the shard data to be verified.

[0096] The slave storage device can search whether there is a slave copy of the shard data to be verified in the slave storage device according to the version number of the shard data to be verified and the version numbers of each shard data stored in the slave storage device.

[0097] S703: When it is determined that a slave copy is stored, a data verification response is generated according to the slave copy of the shard data to be verified and sent to the primary storage device, so that the primary storage device determines the data verification result of the shard data to be verified according to the data verification response.

[0098] If so, the slave storage device can retrieve the corresponding data content from the slave copy according to the partial data content of the master copy in the data verification request, perform data encapsulation to form a data verification response, and feed the data verification response back to the master storage device.

[0099] In an exemplary embodiment, please refer to FIG. 8 , step S702 , determining whether a slave copy of the shard data to be verified is stored based on the master copy of the shard data to be verified, including steps S801 to S803 .

[0100] S801: Obtain the version number of each stored shard data.

[0101] First, the version numbers of all shard data stored in the storage device can be read.

[0102] S802: Search for the version number of each stored shard data according to the version number of the shard data to be verified corresponding to the primary copy.

[0103] The version number of each shard data is compared with the version number of the shard data to be verified in the data verification request one by one to find out whether there is a shard data with the same version number as the shard data to be verified.

[0104] S803: In the case of having the same version number, determine the slave replica storing the shard data to be verified.

[0105] If so, the copy of the shard data corresponding to the same version number is determined as the slave copy of the shard data to be verified.

[0106] In an exemplary embodiment, the present application provides a distributed block storage system, the system comprising multiple storage devices, wherein a master storage device among the multiple storage devices is used to store a master copy of the shard data to be verified, and is used to verify the shard data using any of the above-mentioned data verification methods applied to the master storage device, and a slave storage device among the multiple storage devices is used to store a slave copy of the shard data to be verified, and executes any of the above-mentioned data verification methods applied to the slave storage device.

[0107] In an exemplary embodiment, the present application provides a data repair method, which is applied to a control device in the distributed block storage system of the above embodiment, wherein the control device is respectively connected to multiple storage devices in the distributed block storage system; please refer to Figure 9, the data repair method includes steps S901 to S903.

[0108] S901: Obtain current status information of each storage device.

[0109] The control device can monitor the status of each storage device in the distributed block storage system in real time. The status of the storage device may include normal, offline, etc.

[0110] S902: Determine a faulty storage device among the storage devices according to current status information of each storage device.

[0111] Since the copies of the shard data stored on the storage device will be lost after the storage device goes offline, when an offline storage device is found, the control device can mark the offline storage device as a faulty storage device.

[0112] S903: Determine target storage devices corresponding to each slice data in the faulty storage device according to normal storage devices among the multiple storage devices, so as to write each slice data in the faulty storage device into each target storage device.

[0113] Afterwards, the control device needs to reallocate storage devices for each slice data stored in the failed storage device, that is, the control device needs to determine the target storage device corresponding to each slice data in the failed storage device from the remaining normal storage devices.

[0114] In an exemplary embodiment, step S903, based on normal storage devices among multiple storage devices, respectively determine the target storage devices corresponding to each shard data in the faulty storage device, so as to write each shard data in the faulty storage device into each target storage device, including steps S1001 to S1003.

[0115] S1001: According to normal storage devices among multiple storage devices, respectively determine the copy type and target storage device corresponding to each shard data in the faulty storage device.

[0116] Since different copies of a shard data need to be stored in different storage devices, the control device needs to exclude the storage devices where the secondary copies of the shard data to be repaired are located from the remaining normal storage devices, determine the target storage devices corresponding to the shard data, and control the modification of the metadata in each storage device after the determination is completed.

[0117] S1002: For any shard data in the failed storage device, if the copy type of the shard data is a slave copy, send a data recovery request to a normal storage device storing a master copy of the shard data, so as to write the slave copy of the shard data to the target storage device.

[0118] In the case where the shard data to be repaired is a slave copy, the control device sends a data recovery request to a normal storage device storing the master copy of the shard data, that is, a normal master storage device. The master storage device determines the shard data to be recovered according to the data recovery request, and determines the target storage device from the remaining normal storage devices according to the modified metadata. The master storage device can read all the data content of the master copy of the shard data to be recovered, encapsulate the data to generate a write request, and send it to the target storage device, so that the target storage device writes the data content of the shard data to be recovered in the idle ordinary data block according to the master copy, and generates a new slave copy of the shard data to be recovered.

[0119] S1003: When the copy type of the shard data is the primary copy, any normal storage device storing a secondary copy of the shard data is determined as the primary storage device of the shard data, and a data recovery request is sent to the primary storage device of the shard data to write the secondary copy of the shard data to the target storage device.

[0120] In the case where the shard data to be repaired is the primary copy, the control device can determine any normal storage device that stores a slave copy of the shard data as the primary storage device of the shard data, and then send a data recovery request to the updated primary storage device. The primary storage device determines the shard data to be recovered based on the data recovery request, and determines the target storage device from the remaining normal storage devices based on the modified metadata. The primary storage device can read all the data content of the primary copy of the shard data to be recovered, encapsulate the data to generate a write request, and send it to the target storage device, so that the target storage device writes the data content of the shard data to be recovered in the idle ordinary data block according to the primary copy, and generates a new slave copy of the shard data to be recovered.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiments of the present application also provide a data verification device for implementing the data verification method involved above, and a data repair device for implementing the data repair method involved above. The implementation solutions provided by these devices to solve the problems are similar to the implementation solutions recorded in the above methods, so the specific limitations in one or more data verification device embodiments and data repair device embodiments provided below can refer to the limitations of the data verification method and data repair method above, and will not be repeated here.

[0123] In an exemplary embodiment, the present application provides a data verification device, which is applied to a primary storage device, and the data verification device includes:

[0124] The first execution module is used to send a data verification request to a slave storage device of the shard data to be verified according to the master copy of the shard data to be verified; wherein the slave storage device stores the slave copy of the shard data to be verified.

[0125] The first receiving module is used to receive a data verification response sent from a storage device; wherein the data verification response is generated from the storage device according to the replica and the data verification request.

[0126] The first determination module is used to determine the data verification result of the shard data to be verified based on the primary copy and the data verification response.

[0127] In one embodiment, the shard data to be verified includes a master copy and at least two slave copies, the master copy and the at least two slave copies of the shard data to be verified are stored in different storage devices, respectively, the data verification response includes the slave copy stored in the storage device, and the first determination module includes:

[0128] The first determination submodule is used to determine that the data verification result of the shard data to be verified is abnormal in the master copy when the slave copies are the same and the master copy is different from the slave copy.

[0129] The second determination submodule is used to determine that the data verification result of the shard data to be verified is that the target slave copy is abnormal when the target slave copy among at least two slave copies is different from the remaining slave copies and the master copy, and the master copy is the same as the remaining slave copies.

[0130] In one embodiment, the data verification device further includes:

[0131] The first calibration module is used to calibrate the data of the master copy according to the slave copy when the master copy is abnormal.

[0132] The second calibration module is used to perform data calibration on the target slave replica according to the master replica when the target slave replica is abnormal.

[0133] In one embodiment, the primary storage device includes metadata and shard data to be verified, and the first execution module includes:

[0134] The third determination submodule is used to determine the slave storage device of the shard data to be verified based on the metadata and the master copy of the shard data to be verified; wherein the metadata includes a mapping relationship between the copy of the shard data to be verified and the storage device.

[0135] The first execution submodule is used to generate a data verification request according to the master copy and send it to the slave storage device.

[0136] In one embodiment, the primary storage device includes a plurality of data blocks; among the plurality of data blocks, the metadata data block is used to store metadata, and the target data block is used to store the primary copy of the shard data to be verified; the first execution submodule includes:

[0137] The first reading unit is used to read multiple data clusters of the target data block to obtain the master copy of the shard data to be verified.

[0138] The first encapsulation unit is used to encapsulate the acquired master copy and generate a data verification request.

[0139] The first execution unit is used to send a data verification request to the slave storage device.

[0140] In one embodiment, the data verification device further includes:

[0141] The second receiving module is used to receive a data recovery request sent by the control device.

[0142] The second determination module is used to determine the shard data to be restored and the target storage device according to the data recovery request.

[0143] The second execution module is used to send a write request to the target storage device according to the master copy of the shard data to be restored, so as to write the slave copy of the shard data to be restored in the target storage device.

[0144] In one embodiment, the present application further provides a data verification device, which is applied to a slave storage device, and the data verification device includes:

[0145] The third receiving module is used to receive a data verification request sent by the primary storage device; wherein the data verification request includes the primary copy of the shard data to be verified.

[0146] The third determination module is used to determine whether a slave copy of the shard data to be verified is stored based on the master copy of the shard data to be verified.

[0147] The third execution module is used to generate a data verification response based on the slave copy of the shard data to be verified and send it to the main storage device when it is determined that a slave copy is stored, so that the main storage device determines the data verification result of the shard data to be verified based on the data verification response.

[0148] In one embodiment, the third determination module includes:

[0149] The first acquisition submodule is used to obtain the version number of each stored shard data.

[0150] The first search submodule is used to search for the version number of each stored shard data according to the version number of the shard data to be verified corresponding to the primary copy.

[0151] The fourth determination submodule is used to determine the slave copy storing the shard data to be verified when having the same version number.

[0152] In one embodiment, the present application provides a data repair device, which is applied to a control device in the distributed block storage system of the above embodiment, wherein the control device is respectively connected to multiple storage devices in the distributed block storage system; the data repair device includes:

[0153] The first acquisition module is used to acquire current status information of each storage device.

[0154] The fourth determining module is used to determine a faulty storage device among the storage devices according to the current status information of each storage device.

[0155] The fifth determination module is used to determine the target storage devices corresponding to each slice data in the faulty storage device according to the normal storage devices among the multiple storage devices, so as to write each slice data in the faulty storage device into each target storage device.

[0156] In one embodiment, the fifth determination module includes:

[0157] The fifth determination submodule is used to determine the copy type and target storage device corresponding to each shard data in the faulty storage device according to normal storage devices among the multiple storage devices.

[0158] The second execution submodule is used to send a data recovery request to a normal storage device storing a master copy of any shard data in the faulty storage device, if the copy type of the shard data is a slave copy, so as to write the slave copy of the shard data to the target storage device.

[0159] The third execution submodule is used to determine any normal storage device storing a slave copy of the shard data as the primary storage device of the shard data when the copy type of the shard data is a primary copy, and send a data recovery request to the primary storage device of the shard data to write the slave copy of the shard data to the target storage device.

[0160] Each module in the above-mentioned devices can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0161] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the data verification method described in any of the above embodiments applied to the main storage device, and / or the steps of the data verification method described in any of the above embodiments applied to the slave storage device, and / or the steps of the data repair method described in any of the above embodiments applied to the control device are implemented.

[0162] The computer device may be a terminal, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data verification method and / or a data repair method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0163] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In one embodiment, the present application also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the computer program implements the steps of the data verification method described in any of the above-mentioned embodiments applied to the main storage device, and / or the steps of the data verification method described in any of the above-mentioned embodiments applied to the slave storage device, and / or the steps of the data repair method described in any of the above-mentioned embodiments applied to the control device.

[0165] In one embodiment, the present application also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the data verification method described in any of the above-mentioned embodiments applied to the main storage device, and / or the steps of the data verification method described in any of the above-mentioned embodiments applied to the slave storage device, and / or the steps of the data repair method described in any of the above-mentioned embodiments applied to the control device.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0167] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0169] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data verification method, characterized in that: Applied to a primary storage device, the method comprises: Sending a data verification request to a slave storage device of the shard data to be verified according to the master copy of the shard data to be verified; wherein the slave storage device stores a slave copy of the shard data to be verified; Receiving a data verification response sent by the slave storage device; wherein the data verification response is generated by the slave storage device according to the slave copy and the data verification request; The data verification result of the shard data to be verified is determined according to the primary copy and the data verification response.

2. The data verification method according to claim 1, characterized in that: The shard data to be verified includes a master copy and at least two slave copies, the master copy and at least two slave copies of the shard data to be verified are stored in different storage devices respectively, the data verification response includes the slave copy stored in the slave storage device, and determining the data verification result of the shard data to be verified according to the master copy and each of the data verification responses includes: When the slave copies are the same and the master copy is different from the slave copy, determining that the data verification result of the shard data to be verified is that the master copy is abnormal; When the target slave replica among at least two of the slave replicas is different from the remaining slave replicas and the master replica, and the master replica is the same as the remaining slave replicas, it is determined that the data verification result of the shard data to be verified is that the target slave replica is abnormal.

3. The data verification method according to claim 2, characterized in that: The method further comprises: When the master copy is abnormal, calibrating data of the master copy according to the slave copy; In the case that the target slave copy is abnormal, a data repair request is sent to the slave storage device where the target slave copy is located, so that the slave storage device where the target slave copy is located performs data calibration on the target slave copy according to the master copy.

4. The data verification method according to any one of claims 1 to 3, characterized in that: The primary storage device includes metadata and the shard data to be verified, and sending a data verification request to a slave storage device of the shard data to be verified based on the primary copy of the shard data to be verified includes: Determine a slave storage device of the shard data to be verified according to the metadata and the master copy of the shard data to be verified; wherein the metadata includes a mapping relationship between the copy of the shard data to be verified and the storage device; A data verification request is generated according to the master copy and sent to the slave storage device.

5. The data verification method according to claim 4, characterized in that: The primary storage device includes a plurality of data blocks; among the plurality of data blocks, the metadata data block is used to store the metadata, and the target data block is used to store the primary copy of the shard data to be verified; The step of generating a data verification request according to the master copy and sending the request to the slave storage device includes: Reading multiple data clusters of the target data block to obtain a master copy of the shard data to be verified; Encapsulating the obtained master copy and generating the data verification request; A data verification request is sent to the slave storage device.

6. The data verification method according to any one of claims 1 to 3, characterized in that: The method further comprises: Receiving a data recovery request sent by a control device; Determine the shard data to be restored and the target storage device according to the data recovery request; A write request is sent to the target storage device according to the primary copy of the shard data to be restored, so as to write the secondary copy of the shard data to be restored in the target storage device.

7. A data verification method, characterized in that: Applied to a slave storage device, the method comprises: Receive a data verification request sent by a primary storage device; wherein the data verification request includes a primary copy of the shard data to be verified; Determine whether a slave copy of the shard data to be verified is stored according to the master copy of the shard data to be verified; When it is determined that the slave copy is stored, a data verification response is generated based on the slave copy of the shard data to be verified and sent to the primary storage device, so that the primary storage device determines the data verification result of the shard data to be verified based on the data verification response.

8. The data verification method according to claim 7, characterized in that: The determining whether a slave copy of the shard data to be verified is stored according to the master copy of the shard data to be verified includes: Get the version number of each shard data stored; Searching for the version number of each stored shard data according to the version number of the shard data to be verified corresponding to the primary copy; In the case of having the same version number, determine the slave copy storing the shard data to be verified.

9. A distributed block storage system, characterized in that: The system includes multiple storage devices, wherein a master storage device among the multiple storage devices is used to store a master copy of the shard data to be verified, and is used to verify the shard data using the data verification method described in any one of claims 1 to 6, and a slave storage device among the multiple storage devices is used to store a slave copy of the shard data to be verified, and execute the data verification method described in any one of claims 7 or 8.

10. A data repair method, characterized in that: A control device applied to a distributed block storage system according to claim 9, wherein the control device is respectively connected to a plurality of storage devices in the distributed block storage system; the method comprising: Obtaining current status information of each of the storage devices; Determining a faulty storage device among the storage devices according to current status information of the storage devices; According to normal storage devices among the plurality of storage devices, target storage devices corresponding to each slice data in the faulty storage device are respectively determined, so that each slice data in the faulty storage device is correspondingly written into each target storage device.

Images