Data verification methods, devices, computer equipment and storage media

Abstract

This invention provides a data verification method, apparatus, computer equipment, and storage medium. The method includes: comparing first copy data and second copy data to obtain a first comparison result, wherein the first copy data and the second copy data are backups of each other and stored on different or the same storage nodes in a distributed storage system; generating a first checksum for the first copy data based on the first comparison result being different; and determining that the first copy data is correct and the second copy data is incorrect based on the first checksum being the same as a pre-stored checksum. This invention discovers inconsistencies between copies by comparing the first copy data and the second copy data, and uses the first checksum and a pre-stored checksum to determine the correct copy data. Therefore, this invention can quickly and efficiently determine the accurate copy in distributed storage, ensuring accurate data is obtained from the distributed storage system.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and more specifically, it provides a data verification method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Distributed storage refers to distributing data across multiple independent devices, employing a scalable distributed storage system architecture. It utilizes multiple storage servers to share the storage load and location servers to locate stored information. Therefore, distributed storage not only improves system reliability, availability, and access efficiency but also facilitates expansion. However, users often encounter inconsistencies between data writes and reads due to potential mechanical hard drive failures; this problem is known as silent error.

[0003] Silent errors mainly include three types: bit corruption, phantom writes, and misdirected reads / writes. Bit corruption refers to the spontaneous loss or change of charge in a region of the disk, where the stored charge indicates that the stored data will be deleted, corrupted, or rendered unreadable. Phantom writes refer to a write operation being performed, but no data is actually written to the hard drive (no write). Misdirected reads / writes refer to reading data from the wrong disk location (read error). For multiple replicas in distributed storage, misdirected reads only cause temporary data read errors, which can be corrected by adjusting the data access address to ensure accurate data retrieval. However, for bit corruption or phantom writes, it is impossible to distinguish which replica is correct among multiple replicas. Conventional solutions require manual troubleshooting to determine the correct replica, which is not only very inefficient but also difficult to implement when dealing with large amounts of data. Summary of the Invention

[0004] To address the problem that existing technologies struggle to determine the accurate copy from multiple copies of distributed storage in the event of bit corruption or disk phantom writes, this invention provides a data verification method, apparatus, computer equipment, and storage medium to effectively determine the accurate copy of distributed storage.

[0005] To achieve the above-mentioned technical objectives, the present invention provides a data verification method applied to storage nodes in a distributed storage system; the data verification method may include, but is not limited to, at least one of the following steps.

[0006] The first replica data is compared with the second replica data to obtain a first comparison result; wherein the first replica data and the second replica data are backups of each other, and the first replica data and the second replica data are stored on different or the same storage nodes in the distributed storage system.

[0007] Based on the fact that the first comparison result is different, a first check code for the first copy of the data is generated.

[0008] Based on the fact that the first verification code is the same as the pre-stored verification code, it is determined that the first copy data is the correct copy data and the second copy data is the incorrect copy data.

[0009] In a possible embodiment of the present invention, the data verification method may further include:

[0010] Based on the difference between the first verification code and the pre-stored verification code, a second verification code is generated for the second copy of the data.

[0011] Based on the fact that the second verification code is the same as the pre-stored verification code, it is determined that the second copy data is the correct copy data and the first copy data is the incorrect copy data.

[0012] In a possible embodiment of the present invention, the data verification method may further include:

[0013] Based on the fact that the first verification code is different from the pre-stored verification code and the second verification code is different from the pre-stored verification code, the first data block contained in the first copy data is compared with the second data block contained in the second copy data to obtain a second comparison result of the first data block and the second data block.

[0014] The target data block is repaired based on the second comparison result; the target data block includes at least one first data block and / or at least one second data block.

[0015] Generate a new checksum for the repaired target data block.

[0016] The repair was confirmed to be successful if the new checksum matches the pre-stored checksum of the target data block.

[0017] In a possible embodiment of the present invention, repairing the target data block according to the second comparison result may include: using the erroneous first data block and / or the erroneous second data block as the target data block according to the second comparison result; and repairing the data bits in the target data block.

[0018] In one embodiment of the present invention, repairing the data bits in the target data block includes:

[0019] The first data bits contained in the erroneous first data block are compared with the second data bits contained in the erroneous second data block to obtain a third comparison result.

[0020] Based on the third comparison result, the erroneous data bits in the target data block are determined.

[0021] Reverse multiple consecutive erroneous data bits, and / or reverse erroneous data bits one by one.

[0022] In a possible embodiment of the present invention, the data verification method may further include: repairing erroneous copy data with correct copy data so that the first copy data is identical to the second copy data.

[0023] In an embodiment of the present invention, the data verification method further includes: obtaining the physical address of the first copy data or the second copy data; obtaining the pre-stored verification code according to the mapping relationship between the physical address and the pre-stored verification code; the pre-stored verification code is stored in the verification area of ​​the solid-state drive, the solid-state drive is a solid-state drive that supports end-to-end data protection, and the type of the pre-stored verification code is a data integrity field or a data integrity extended type.

[0024] In an embodiment of the present invention, before comparing the first copy data with the second copy data, the data verification method further includes: receiving a data verification instruction; traversing the data of each storage volume of the mechanical hard disk based on the data verification instruction to determine the first copy data and the second copy data; and using the cache area of ​​the solid-state drive to cache and accelerate the mechanical hard disk.

[0025] To achieve the above technical objectives, the present invention provides a data verification device applied to a storage node in a distributed storage system; the data verification device includes, but is not limited to, a replica data comparison module, a checksum generation module, and a replica data determination module.

[0026] The replica data comparison module is used to compare the first replica data with the second replica data to obtain a first comparison result between the first replica data and the second replica data; wherein the first replica data and the second replica data are backups of each other, and the first replica data and the second replica data are stored on different or the same storage nodes in the distributed storage system.

[0027] The check code generation module is used to generate a first check code for the first copy of the data based on the fact that the first comparison results are different.

[0028] The copy data determination module is used to determine that the first copy data is correct and the second copy data is incorrect based on the fact that the first verification code is the same as the pre-stored verification code.

[0029] In one embodiment of the present invention, the verification code generation module is used to generate a second verification code for the second copy data based on the difference between the first verification code and the pre-stored verification code; the copy data determination module is used to determine that the second copy data is correct copy data and the first copy data is incorrect copy data based on the fact that the second verification code is the same as the pre-stored verification code.

[0030] In a possible embodiment of the present invention, the data verification device may further include a data block comparison module, a data block repair module, and a repair result judgment module.

[0031] The data block comparison module is used to compare the first data block contained in the first copy data with the second data block contained in the second copy data based on the fact that the first check code is different from the pre-stored check code and the second check code is different from the pre-stored check code, so as to obtain a second comparison result between the first data block and the second data block.

[0032] The data block repair module is used to repair the target data block according to the second comparison result; the target data block includes at least one first data block and / or at least one second data block.

[0033] The repair result judgment module is used to determine that the repair is successful if the new check code is the same as the pre-stored check code of the target data block; the check code generation module is used to generate a new check code for the repaired target data block.

[0034] In one embodiment of the present invention, a data block repair module is used to use an erroneous first data block and / or an erroneous second data block as target data blocks according to a second comparison result, and to repair the data bits in the target data blocks.

[0035] In one embodiment of the present invention, a data block repair module is used to compare the first data bits contained in an erroneous first data block with the second data bits contained in an erroneous second data block to obtain a third comparison result.

[0036] The data block repair module is used to determine the erroneous data bits in the target data block based on the third comparison result.

[0037] The data block repair module is used to reverse multiple consecutive erroneous data bits, and / or to reverse erroneous data bits one by one.

[0038] In one embodiment of the present invention, the data block repair module is further used to repair erroneous copy data using correct copy data, so that the first copy data is identical to the second copy data.

[0039] In some embodiments of the present invention, the data verification device may further include, but is not limited to, a verification code acquisition module.

[0040] The verification code acquisition module is used to obtain the physical address of the first copy data or the second copy data, and to obtain the pre-stored verification code according to the mapping relationship between the physical address and the pre-stored verification code; the pre-stored verification code is stored in the verification area of ​​the solid-state drive; the solid-state drive is a solid-state drive that supports end-to-end data protection, and the pre-stored verification code type is a data integrity field or a data integrity extended type.

[0041] In some embodiments of the present invention, the data verification device includes, but is not limited to, a data verification startup module.

[0042] The data verification startup module is used to receive data verification commands and to traverse the data of each storage volume of the mechanical hard drive based on the data verification commands to determine the first copy data and the second copy data; the cache area of ​​the solid-state drive is used to cache and accelerate the mechanical hard drive.

[0043] To achieve the above-mentioned technical objectives, the present invention may also provide a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the data verification method in any embodiment of the present invention.

[0044] To achieve the above-mentioned technical objectives, the present invention can also provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the data verification method in any embodiment of the present invention.

[0045] The beneficial effects of this invention are as follows: When faced with the problem of difficulty in determining the accurate replica from multiple replicas in distributed storage in situations involving bit corruption or disk phantom writes, this invention identifies replica inconsistencies by comparing the first replica data and the second replica data. Based on this, it determines that the first replica data is correct and the second replica data is incorrect by comparing the first checksum of the generated first replica data with a pre-stored checksum. Therefore, this invention can quickly and efficiently determine the accurate replica in distributed storage, ensuring end-to-end data consistency based on the determined accurate replica, thereby guaranteeing accurate data retrieval from the distributed storage system, improving user experience, and resulting in higher user satisfaction. Attached Figure Description

[0046] Figure 1 A schematic diagram illustrating one possible implementation of the data verification method in one or more embodiments of the present invention is shown.

[0047] Figure 2 A schematic diagram illustrating another implementation of the data verification method in one or more embodiments of the present invention is shown.

[0048] Figure 3 A schematic diagram illustrating another embodiment of the data verification method in one or more embodiments of the present invention is shown.

[0049] Figure 4 A schematic diagram of a distributed storage structure with a cache acceleration architecture is shown in one or more embodiments of the present invention.

[0050] Figure 5 This diagram illustrates the cache mapping relationship between cache lines and core lines in one or more embodiments of the present invention.

[0051] Figure 6 A schematic diagram of one possible implementation of the data verification device according to one or more embodiments of the present invention is shown.

[0052] Figure 7 A schematic diagram of another embodiment of the data verification device in one or more embodiments of the present invention is shown.

[0053] Figure 8 A schematic diagram of the internal structure of a computer device is shown in one or more embodiments of the present invention. Detailed Implementation

[0054] The data verification method, apparatus, computer equipment, and storage medium provided by the present invention will be explained and described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, and can be combined Figure 2 and Figure 3 One or more embodiments of the present invention provide a data verification method, which is applied to the field of distributed storage technology, specifically to the storage nodes of a distributed storage system. The data verification method may include, but is not limited to, at least one of the following steps.

[0056] Step 100: Compare the first replica data with the second replica data to obtain a first comparison result. The first and second replica data serve as backups of each other and are stored on different or the same storage nodes in the distributed storage system. Both the first and second replica data are replicas of the distributed storage system and can function as two replicas within the cluster. Alternatively, the first and second replica data can function as three replicas of the distributed storage system, with the first replica serving as two replicas and the second replica as one replica, or vice versa. It should be understood that, in the absence of any errors or faults, the different replicas of the distributed storage system are identical. This embodiment uses a two-replica distributed storage system as an example. The first and second replica data constitute the two replicas of the distributed storage system. In this embodiment, the first and second replica data can be distributed on different storage nodes or on the same storage node.

[0057] Step 200: Based on the first comparison result being different, generate a first checksum for the first copy data. A different first comparison result indicates that the first copy data and the second copy data are inconsistent; that is, the two copies in this embodiment of distributed storage are different, and the accurate copy needs to be determined. In this embodiment, the type of the first checksum is the same as the pre-stored checksum type, such as DIF (Data Integrity Field) or DIX (Data Integrity Extensions). For example, the DIF or DIX checksum can be an 8-byte contiguous data appended after the 512-byte data area on the hard disk; this 8-byte data is called (PI, Protection Information). For the DIF-Data Integrity Field type, the PI information is directly set after the user data, and the user data and PI share a buffer; for the DIX-Data Integrity Extensions type, the PI information is a separate buffer, meaning two independent buffers are needed: one for storing user data and the other for storing PI information. For example, if the data block size is 512 bytes and PI is enabled, if 8 data blocks are stored, a buffer of (512+8)×8 bytes is needed for DIF, and a buffer of 512×8 bytes and an 8×8 byte PI buffer are needed for DIX.

[0058] In this embodiment, taking a first replica data (chunk1) size of 1M bytes, a data block size of 512 bytes, and a cache line size of 64K bytes as an example, the first replica data consists of 16 cache lines and requires traversing 2048 data blocks. Each data block corresponds to a checksum, so the number of first checksums generated for the first replica data in this embodiment can be 2048. Each first checksum can be, for example, ci, i = 1, 2, 3, ..., 2048.

[0059] In this invention, if the first comparison result is different, a first checksum of the first copy data is generated; if the first comparison result is the same, the data consistency check passes, and the process continues with the next copy data (chunk), i.e., returning to step 100 to check the remaining copy data. Clearly, this invention only calculates the checksum when the comparison results between the first and second copy data are different. Therefore, this invention has significant advantages such as lower resource consumption, higher execution efficiency, and more convenient implementation.

[0060] Step 300: Based on the first checksum being the same as the pre-stored checksum, it is determined that the first replica data is correct and the second replica data is incorrect. Specifically, this invention uses the pre-stored checksum as the criterion for determining the accuracy of replica data. If the generated checksum of the current replica data is the same as the pre-stored checksum, it indicates that the current replica data is correct; if the generated checksum of the current replica data is different from the pre-stored checksum, it indicates that the current replica data is incorrect. This invention can quickly and accurately determine that the first replica data is correct based on the first checksum being the same as the pre-stored checksum, and that the second replica data is incorrect if the first replica data is different from the second replica data. Therefore, this invention can effectively determine the accurate replicas in distributed storage, solving the problem of conventional technologies being unable to automatically distinguish which replica among multiple replicas is correct.

[0061] The data verification method in one or more embodiments of the present invention may further include: repairing erroneous copy data using correct copy data. In this embodiment, the correct first copy data is used to repair erroneous second copy data so that the second copy data is the same as the first copy data. In specific implementation, the present invention repairs the data by overwriting the second copy data with the first copy data.

[0062] like Figure 4 As shown, the data verification method in this embodiment of the invention may further include, before comparing the first copy data and the second copy data: receiving a data verification instruction, and traversing the data of each storage volume of the mechanical hard disk based on the data verification instruction to determine the first copy data and the second copy data; the cache area of ​​the solid-state drive is used to cache and accelerate the mechanical hard disk. In specific implementation, the storage engine processing module issues a data verification instruction according to a preset period to trigger data consistency verification at a preset period. The verification device of the present invention can receive the data verification instruction according to the preset period. The data verification instruction is used to start the data traversal process, specifically traversing the copy data (chunk) of each storage volume in the node one by one. The traversed copy data may include the first copy data chunk1 and the second copy data chunk1' involved in the present invention. The first copy data and the second copy data of the present invention may be stored in different nodes or different mechanical hard disks. Figure 4Three storage nodes are shown, but the number of nodes in a specific implementation can be set according to actual needs. Each storage node includes n hard disks, where n ≥ 1 and is an integer. In this embodiment, each storage node is equipped with a solid-state drive (SSD). The SSD includes a cache area and a checksum area. The cache area is used to cache data to be stored on the mechanical hard disk, and the checksum area is used to store pre-stored checksums, which can be of type DIF or DIX. This invention simultaneously implements the functions of a cache disk and a checksum storage disk using SSDs without adding new hardware media, thus reducing the construction cost of a distributed storage system.

[0063] like Figure 4 and Figure 5 As shown, the data verification method in this embodiment of the invention may further include a process of obtaining a pre-stored verification code: obtaining the physical address of the first copy data or the second copy data, and obtaining the pre-stored verification code according to the mapping relationship between the physical address and the pre-stored verification code; the pre-stored verification code is stored in the verification area of ​​the solid-state drive, the solid-state drive is a solid-state drive that supports end-to-end data protection, and the type of the pre-stored verification code is a data integrity field or a data integrity extended type. Figure 5 The mapping relationships shown can be, for example, a cache mapping between core line 5 and cache line 1, a cache mapping between core line 1 and cache line 2, a cache mapping between core line 6 and cache line 3, a cache mapping between core line 2 and cache line 4, a cache mapping between core line 9 and cache line 5, and a cache mapping between core line 3 and cache line 6, but are not limited to these. In this embodiment of the invention, the physical address of the replica data is obtained by querying the metadata of the replica data. Specifically, the storage location of the first replica data chunk1 can be determined by querying the metadata of the first replica data chunk1 in the distributed storage. This storage location is, for example, the node 1 disk sdb 0x10000 location; then, the SSD checksum corresponding to the physical address of the replica data is accessed, and the pre-stored checksum corresponding to the index at the node 1 disk sdb 0x10000 location is used. For the first replica data (chunk1) of 1M bytes, the CRC16 value of the checksum can be ai, i = 1, 2, 3, ..., 2048. In specific implementation, this embodiment can obtain the pre-stored verification code through an application programming interface (API) call. Determining whether the first verification code is the same as the pre-stored verification code in this embodiment may include: determining whether ci == ai is true; if true, it indicates that the first verification code is the same as the pre-stored verification code, and the first copy data is determined to be correct copy data while the second copy data is incorrect copy data.

[0064] As a preferred embodiment, the pre-stored checksum, first checksum, and second checksum of the present invention can be generated using an SSD controller chip. That is, the present invention can accelerate checksum calculation through SSD hardware capabilities, enabling automatic generation and storage of checksums based on the capabilities of solid-state drive chips. Compared to the conventional method of calculating checksums using the CPU, which consumes significant CPU resources, the technical solution provided by the present invention can bring higher performance to distributed storage systems.

[0065] The distributed storage system of this invention may include an IO (Input / Output) engine processing module and a cache processing module. The IO engine processing module can respond to IO requests from storage clients, distribute multiple replica IO requests to designated nodes, and periodically initiate data verification. The cache processing module can cache data in the local node's cache area and save pre-stored verification codes for local node data blocks in the local node's verification area. In specific implementation, the data storage process may include an initialization process and a verification code storage process during the IO write process. During initialization, the SSD is divided into the cache area and verification area as described above. The cache area serves as the cache area for the local node's mechanical hard drives, and the verification area serves as the pre-stored verification code for the data blocks of the local node's mechanical hard drives. In this embodiment, for example, the ratio of data blocks to verification codes can be 512:8 = 64:1, and the storage space of the verification area should be at least 1 / 64 of the total capacity of all mechanical hard drives in the node. If the node is allowed to add mechanical hard drives, the SSD needs to reserve space for the verification area. The initialization process can be performed by clustering the storage IO engine according to the cache area and multiple mechanical hard drive architecture to configure multiple nodes. During the checksum storage process, firstly, the storage client accesses the storage system and issues write IO to the storage IO engine. The storage IO engine queries the metadata of the corresponding replica data (chunk) for the IO, such as obtaining the two replica node numbers 1 and 2, and distributing the two replica IOs to the specified nodes 1 and 2. Secondly, the cache processing module first writes the data to the SSD cache area on node 1, and then writes it to the hard disk storage space. When writing to the SSD cache, the hardware feature of SSD supporting DIF or DIX can be utilized to automatically generate a pre-stored checksum for each piece of data written to the SSD through the SSD controller chip. This checksum can be, for example, a CRC16 (Cyclic Redundancy Check) checksum, and the pre-stored checksum is stored in the SSD checksum area. The pre-stored checksum can be retrieved later through the API. Before the cache is written to disk or before the cache is evicted, the current cache line is divided into 512-byte blocks, and the SSD driver uses DIF or DIX to process the data. The API is used to determine the pre-stored checksum ai corresponding to a data block of size 512 bytes, where i is in units of 512 bytes; in node 1, the cached metadata in memory is searched to obtain the physical disk (e.g., hard disk drive) address bj corresponding to the current cache line, for example... Figure 5Cache lines 1 to 6 and core lines 1 to 9 can specifically be backend storage. The core lines store physical addresses bj, where j is the size of a cache line; in this embodiment, the cache line size can be 64KB. Then, using the physical address bj as an index, ai is sequentially saved to the verification area. Each cache line corresponds to 64KB / 512 = 125 checksums. Finally, the cache processing module can be used to write the cached IO data to disk at a designated node 1, completing the data storage to the mechanical hard drive. For node 2, the checksum storage process and data storage process are similar to those of node 1, and will not be described further. The above method implements the writing process of two copies, realizing the process of writing the first copy data and the second copy data to the corresponding node's mechanical hard drive in this invention.

[0066] like Figure 2 As shown, if the first verification code is different from the pre-stored verification code, the data verification method of this embodiment may further include steps 400 and 500.

[0067] Step 400: Based on the difference between the first check code and the pre-stored check code, a second check code for the second copy data is generated. In this embodiment of the invention, the type of the second check code is also the same as the pre-stored check code type, for example, it can be DIF or DIX. Therefore, in the case where the first copy data chunk1 fails verification, this invention continues to verify the second copy data chunk1' to determine whether the second copy data chunk1' is correct data.

[0068] Step 500: Based on the fact that the second verification code matches the pre-stored verification code, it is determined that the second copy data is the correct copy data and the first copy data is the incorrect copy data. This invention is used in distributed storage systems such as Ceph to respond to user data requests by quickly determining the correct copy data.

[0069] The data verification method in this embodiment of the invention may further include: repairing erroneous copy data with correct copy data, specifically repairing erroneous first copy data with correct second copy data, so that the first copy data is the same as the second copy data; in specific implementation of this embodiment, the repair can be performed by overwriting the first copy data with correct second copy data.

[0070] like Figure 3 As shown, if the first verification code of the present invention is different from the pre-stored verification code and the second verification code is different from the pre-stored verification code, the data verification method of the present invention further includes steps 600 and 700.

[0071] Step 600: Based on the fact that the first checksum is different from the pre-stored checksum and the second checksum is different from the pre-stored checksum, compare the first data block contained in the first copy data with the second data block contained in the second copy data to obtain a second comparison result of the first data block and the second data block. It can be seen that the present invention can also perform exploratory repair processing on the copy data when both the first checksum and the second checksum are different from the pre-stored checksum, that is, when both the first copy data chunk1 and the second copy data chunk1' are inaccurate.

[0072] Step 700: Repair the target data block according to the second comparison result; the target data block includes at least one first data block and / or at least one second data block. It is evident that the present invention can repair the first data block contained in the first copy data, or repair the second data block contained in the second copy data, or repair both the first data block contained in the first copy data and the second data block contained in the second copy data. Compared to the traditional method of leaving data anomaly problems to users to resolve manually, the present invention can automatically repair erroneous copy data, ensuring that users can read accurate data.

[0073] This invention can specifically repair erroneous data blocks. Repairing the target data block based on the second comparison result includes: using the erroneous first data block and / or the erroneous second data block as the target data block based on the second comparison result; and repairing the data bits in the target data block. Taking a replica data chunk size of 1MB as an example, each chunk includes 2048 data blocks of 512 bytes each. The 2048 first data blocks contained in the first replica data chunk1 are compared with the 2048 data blocks contained in the second replica data chunk1'. For example, if data block ax in the first replica data chunk1 is found to be different from data block ax' in the second replica data chunk1', that is, the checksum calculated for data block ax is different from the corresponding pre-stored checksum, and the checksum calculated for data block ax' is different from the corresponding pre-stored checksum, then data blocks ax and ax' are determined to be erroneous data blocks (target data blocks). This embodiment performs repair processing on data blocks ax and / or data blocks ax'.

[0074] This embodiment can specifically repair erroneous data bits in erroneous data blocks. The repair of data bits in the target data block in this embodiment may include: comparing the first data bits contained in the erroneous first data block with the second data bits contained in the erroneous second data block to obtain a third comparison result; and reversing the data bits in the target data block based on the third comparison result. Specifically, based on the third comparison result, identical first and second data bits are determined, and different first and second data bits are determined. For identical first and second data bits, their data bits are retained as new data bits, while for different first and second data bits, corresponding repair is performed, and the repair method is reversal processing.

[0075] Specifically, in one or more embodiments of the present invention, the inversion processing of data bits in a target data block based on a third comparison result includes: determining erroneous data bits in the target data block based on the third comparison result; inverting multiple consecutive erroneous data bits; and / or inverting erroneous data bits one by one. The present invention, through its data repair method of inverting multiple consecutive erroneous data bits, can repair multiple data bits with regional errors more quickly and accurately; the present invention, through its repair method of inverting erroneous data bits one by one, can repair individual erroneous data bits more accurately and effectively. Therefore, the present invention can achieve targeted repair of data errors in different situations.

[0076] The data verification method in one or more embodiments of the present invention further includes: generating a new verification code for the repaired target data block, and determining that the repair is successful if the new verification code is the same as a pre-stored verification code of the target data block; otherwise, the repair fails. Based on the above method for determining whether the repair is successful, the present invention can also generate a new verification code for the repaired copy data, and compare the new verification code of the copy data with a pre-stored verification code of the copy data; if they are the same, the repair is successful; otherwise, the repair fails.

[0077] like Figure 6 As shown, based on the same inventive concept as the data verification method, one or more embodiments of the present invention can also provide a data verification device, which is applied in the field of distributed storage technology, specifically applied to the storage nodes of a distributed storage system.

[0078] Corresponding to the data verification method, the data verification device includes, but is not limited to, a copy data comparison module, a check code generation module, and a copy data determination module, as detailed below.

[0079] The replica data comparison module is used to compare the first replica data with the second replica data to obtain a first comparison result between the first replica data and the second replica data; wherein the first replica data and the second replica data are backups of each other, and the first replica data and the second replica data are stored on different or the same storage nodes in the distributed storage system.

[0080] The check code generation module is used to generate a first check code for the first copy of the data based on the fact that the first comparison results are different.

[0081] The copy data determination module is used to determine that the first copy data is correct and the second copy data is incorrect based on the fact that the first verification code is the same as the pre-stored verification code.

[0082] The verification code generation module in this embodiment of the invention can also be used to generate a second verification code for the second copy data based on the difference between the first verification code and the pre-stored verification code. The copy data determination module can also be used to determine that the second copy data is correct copy data and the first copy data is incorrect copy data based on the fact that the second verification code is the same as the pre-stored verification code.

[0083] like Figure 7 As shown, the data verification device in one or more embodiments of the present invention may further include a data block comparison module, a data repair module, and a repair result judgment module.

[0084] The data block comparison module is used to compare a first data block contained in a first copy of data with a second data block contained in a second copy of data, based on the fact that a first checksum differs from a pre-stored checksum and a second checksum differs from a pre-stored checksum, to obtain a second comparison result between the first data block and the second data block. The data block repair module is used to repair a target data block based on the second comparison result; the target data block includes at least one first data block and / or at least one second data block.

[0085] The repair result judgment module is used to determine that the repair is successful if the new check code is the same as the pre-stored check code of the target data block; the check code generation module is used to generate a new check code for the repaired target data block.

[0086] Specifically, the data block repair module is used to use the erroneous first data block and / or the erroneous second data block as target data blocks according to the second comparison result, and to repair the data bits in the target data block.

[0087] More specifically, the data block repair module is used to compare the first data bits contained in the erroneous first data block with the second data bits contained in the erroneous second data block to obtain a third comparison result; the data block repair module is used to reverse the data bits in the target data block according to the third comparison result. The data block repair module of one or more embodiments of the present invention is used to determine the erroneous data bits in the target data block according to the third comparison result; the data block repair module is used to reverse multiple consecutive erroneous data bits, and / or to reverse the erroneous data bits one by one.

[0088] Optionally, the data block repair module in one or more embodiments of the present invention can also be used to repair erroneous copy data using correct copy data, so that the first copy data is the same as the second copy data.

[0089] The data verification device in one or more embodiments of the present invention may further include a verification code acquisition module.

[0090] The verification code acquisition module is used to obtain the physical address of the first copy data or the second copy data, and to obtain the pre-stored verification code according to the mapping relationship between the physical address and the pre-stored verification code. The pre-stored verification code is stored in the verification area of ​​the solid-state drive. The solid-state drive is a solid-state drive that supports end-to-end data protection. The type of the pre-stored verification code is a data integrity field or a data integrity extended type.

[0091] The data verification device in one or more embodiments of the present invention may further include a data verification startup module.

[0092] The data verification startup module is used to receive data verification commands and to traverse the data of each storage volume of the mechanical hard drive based on the data verification commands to determine the first copy data and the second copy data; the cache area of ​​the solid-state drive is used to cache and accelerate the mechanical hard drive.

[0093] like Figure 8 As shown, based on the same inventive concept as the data verification method, one or more embodiments of the present invention can also provide a computer device, which includes a memory and a processor. The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the data verification method in any embodiment of the present invention. The detailed execution process of the data verification method has been described in detail in this specification and will not be repeated here.

[0094] like Figure 8As shown, based on the same inventive concept as the data verification method, one or more embodiments of the present invention may also provide a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the steps of the data verification method in any embodiment of the present invention. The detailed execution process of the data verification method has been described in detail in this specification and will not be repeated here.

[0095] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0096] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0097] In the description of this specification, the references to terms such as "this embodiment," "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and simple improvements made on the substantive content of the present invention should be included within the protection scope of the present invention.

Claims

1. A data verification method, characterized in that, The method is applied to storage nodes in a distributed storage system with a caching acceleration architecture, wherein the storage nodes include hard disk drives (HDDs) and solid-state drives (SSDs); the method includes: The first copy of data and the second copy of data in the mechanical hard disk are compared to obtain a first comparison result of the first copy of data and the second copy of data; wherein the first copy of data and the second copy of data are backups of each other, and the first copy of data and the second copy of data are stored on different or the same storage nodes in the distributed storage system; If the first comparison result is the same, the data consistency check passes; Based on the fact that the first comparison result is different, a first check code for the first copy data is generated; Based on the fact that the first verification code is the same as the pre-stored verification code, it is determined that the first copy data is correct copy data and the second copy data is incorrect copy data; the solid-state drive includes a cache area and a verification area, the cache area is used to cache data to be stored to the mechanical hard drive, and the verification area is used to store the pre-stored verification code; Based on the difference between the first verification code and the pre-stored verification code, a second verification code for the second copy of the data is generated; Based on the fact that the first verification code is different from the pre-stored verification code and the second verification code is different from the pre-stored verification code, the target data block is repaired according to the second comparison result of the first data block contained in the first copy data and the second data block contained in the second copy data. The target data block includes at least one first data block and / or at least one second data block. Repairing the target data block based on the second comparison result includes: Based on the second comparison result, the erroneous first data block and / or the erroneous second data block are used as the target data block; Repair the data bits in the target data block; The repair of data bits in the target data block includes: The first data bits contained in the erroneous first data block are compared with the second data bits contained in the erroneous second data block to obtain a third comparison result; Based on the third comparison result, the erroneous data bits in the target data block are determined; Reverse multiple consecutive erroneous data bits, and / or reverse erroneous data bits one by one.

2. The data verification method according to claim 1, characterized in that, The method further includes: Based on the fact that the second verification code is the same as the pre-stored verification code, it is determined that the second copy data is the correct copy data and the first copy data is the incorrect copy data.

3. The data verification method according to claim 2, characterized in that, The method further includes: Generate a new checksum for the repaired target data block; The repair is confirmed to be successful if the new checksum is the same as the pre-stored checksum of the target data block.

4. The data verification method according to any one of claims 1 to 3, characterized in that, The method further includes: Correct the incorrect replica data using the correct replica data, so that the first replica data is identical to the second replica data.

5. The data verification method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the physical address of the first or second copy of the data; The pre-stored checksum is obtained based on the mapping relationship between the physical address and the pre-stored checksum; the solid-state drive is a solid-state drive that supports end-to-end data protection, and the pre-stored checksum type is a data integrity field or a data integrity extended type.

6. The data verification method according to claim 5, characterized in that, Before comparing the first copy data with the second copy data, the method further includes: Receive data verification command; The data is traversed through the data of each storage volume of the mechanical hard drive based on the data verification instruction to determine the first copy data and the second copy data; the cache area of ​​the solid-state drive is used to cache and accelerate the mechanical hard drive.

7. A data verification device, characterized in that, The device is applied to storage nodes in a distributed storage system with a caching acceleration architecture, the storage nodes including hard disk drives (HDDs) and solid-state drives (SSDs); the device includes: The replica data comparison module is used to compare the first replica data and the second replica data in the mechanical hard disk to obtain a first comparison result between the first replica data and the second replica data. If the first comparison result is the same, the data consistency check passes. The first replica data and the second replica data are backups of each other, and the first replica data and the second replica data are stored on different or the same storage nodes in the distributed storage system. The verification code generation module is used to generate a first verification code for the first copy data based on the fact that the first comparison result is different. The copy data determination module is used to determine that the first copy data is correct copy data and the second copy data is incorrect copy data based on the fact that the first verification code is the same as the pre-stored verification code; the solid-state drive includes a cache area and a verification area, the cache area is used to cache data to be stored to the mechanical hard drive, and the verification area is used to store the pre-stored verification code; The verification code generation module is further configured to generate a second verification code for the second copy data based on the difference between the first verification code and the pre-stored verification code. The data block repair module is used to repair a target data block based on a second comparison result between a first data block contained in the first copy data and a second data block contained in the second copy data, since the first check code is different from the pre-stored check code and the second check code is different from the pre-stored check code; the target data block includes at least one first data block and / or at least one second data block. The data block repair module is used to take the erroneous first data block and / or the erroneous second data block as the target data block according to the second comparison result, and to repair the data bits in the target data block; The data block repair module is used to compare the first data bits contained in the erroneous first data block with the second data bits contained in the erroneous second data block to obtain a third comparison result; The data block repair module is used to determine the erroneous data bits in the target data block based on the third comparison result; The data block repair module is used to reverse multiple consecutive erroneous data bits, and / or to reverse erroneous data bits one by one.

8. The data verification device according to claim 7, characterized in that, The copy data determination module is used to determine that the second copy data is correct copy data and the first copy data is incorrect copy data based on the fact that the second verification code is the same as the pre-stored verification code.

9. The data verification device according to claim 8, characterized in that, The device further includes: The data block comparison module is used to compare the first data block contained in the first copy data with the second data block contained in the second copy data based on the fact that the first check code is different from the pre-stored check code and the second check code is different from the pre-stored check code, so as to obtain a second comparison result between the first data block and the second data block; The repair result judgment module is used to determine that the repair is successful if the new check code is the same as the pre-stored check code of the target data block; the check code generation module is used to generate a new check code for the repaired target data block.

10. The data verification device according to any one of claims 7 to 9, characterized in that, The device further includes: The data block repair module is used to repair erroneous copy data using correct copy data, so that the first copy data is identical to the second copy data.

11. The data verification device according to any one of claims 7 to 9, characterized in that, The device further includes: The verification code acquisition module is used to acquire the physical address of the first copy data or the second copy data, and to acquire the pre-stored verification code according to the mapping relationship between the physical address and the pre-stored verification code; the solid-state drive is a solid-state drive that supports end-to-end data protection, and the pre-stored verification code type is a data integrity field or a data integrity extended type.

12. The data verification device according to claim 11, characterized in that, The device further includes: The data verification startup module is used to receive data verification instructions and to traverse the data of each storage volume of the mechanical hard disk based on the data verification instructions to determine the first copy data and the second copy data; the cache area of ​​the solid-state drive is used to cache and accelerate the mechanical hard disk.

13. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, cause the processor to perform the steps of the data verification method as described in any one of claims 1 to 6.

14. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the steps of the data verification method as claimed in any one of claims 1 to 6.

Images