Data verification method and system
By generating fast consistency check commands through the RAID controller, the storage device calculates and sends data values, and the RAID controller compares the calculated values, the problem of excessive bandwidth consumption by the RAID card in data consistency verification is solved, thus improving efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DAPUSTOR CORP
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-05
AI Technical Summary
In existing technologies, RAID cards perform a large number of invalid read and write operations when performing data consistency checks, resulting in excessive consumption of bandwidth resources between the RAID and SSD, which reduces the efficiency of data consistency checks.
The RAID controller obtains information about the storage device, generates a fast consistency check command, calculates data based on the command, and sends the calculated value to the RAID controller. The RAID controller compares the calculated value, reducing data transfer volume and improving consistency check efficiency.
This reduces the amount of data transfer between the RAID controller and storage devices, lowers bandwidth utilization, and improves the efficiency of consistency checks across multiple storage devices.
Smart Images
Figure CN121979448A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of storage technology, and in particular to a data verification method and system thereof. Background Technology
[0002] Redundant Array of Independent Disks (RAID) technology combines multiple disks to form a unified logical storage unit, thereby optimizing and improving the performance, reliability, and capacity of the storage system. Consistency Check (CC) is an advanced maintenance function of the RAID array controller, which ensures the consistency of data across multiple disks by pre-checking the data on the array.
[0003] Currently, when RAID cards perform data checks on all disks, they typically read data from different disks and compare the data from different disks using the controller inside the RAID card to confirm data consistency. However, performing full read and transfer comparisons on valid data that has already been kept consistent and on idle blank areas of the disks generates a large number of invalid read and write operations. This leads to excessive occupation of bandwidth resources between RAID and SSD, increases the centralized computing load on the RAID card, and reduces the efficiency of data consistency verification. Summary of the Invention
[0004] To address the aforementioned technical problems, embodiments of this application provide a data verification method and system that can reduce the amount of data transmission between the RAID controller and multiple storage devices during consistency checks, thereby improving the efficiency of consistency checks on multiple storage devices.
[0005] To address the aforementioned technical problems, the embodiments of this application provide the following technical solutions: In a first aspect, embodiments of this application provide a data verification method applied to a data verification system. The data verification system includes a RAID controller and at least two storage devices communicatively connected to the RAID controller. Each storage device includes a data storage module. The method includes: The RAID controller obtains information about the storage devices and generates fast consistency check commands based on that information. The storage device obtains the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, performs calculations on the data corresponding to the fast consistency check instruction to obtain the calculated value, and sends the calculated value to the RAID controller; The RAID controller acquires the calculated values sent by at least two storage devices, compares the calculated values of the at least two storage devices, and obtains the first verification result.
[0006] In some embodiments, the information of the storage device includes a computing mode, the computing mode includes at least two data integrity verification algorithms, and the fast consistency check instruction includes a computing algorithm, the computing algorithm being one of the at least two data integrity verification algorithms. Before retrieving the data corresponding to the fast consistency check instruction from the data storage module, the method also includes: The storage device initiates a fast consistency check based on the fast consistency check instruction. The calculated values are obtained by performing calculations on the data corresponding to the fast consistency check instructions, including: The calculation algorithm is used to calculate the data corresponding to the fast consistency check instruction to obtain the calculated value.
[0007] In some embodiments, the RAID controller includes a hardware accelerator and a preset comparison program; The calculated values from at least two storage devices are compared to obtain a first verification result, including: If the calculated data length is greater than or equal to the preset data length, the RAID controller activates the hardware accelerator to compare the calculated values of at least two storage devices and obtain the first verification result. If the calculated data length is less than the preset data length, the RAID controller compares the calculated values of at least two storage devices using a preset comparison program to obtain the first verification result.
[0008] In some embodiments, the method further includes: The RAID controller generates a full consistency check command based on a preset period and sends the full consistency check command to the storage devices to obtain the data corresponding to the full consistency check command returned by at least two storage devices. The RAID controller verifies the data corresponding to the full consistency check instructions returned by at least two storage devices to obtain a second verification result.
[0009] In some embodiments, the method further includes, prior to generating a full consistency check instruction: Get configuration information; Based on the configuration information, set the operating parameters of the hardware accelerator; After obtaining the data corresponding to the full consistency check instructions returned by at least two storage devices, the method further includes: Based on the operating parameters of the hardware accelerator, the data corresponding to the full consistency check instructions returned by at least two storage devices is verified to obtain a second verification result.
[0010] Secondly, embodiments of this application provide a data verification system, the system including a RAID controller and at least two storage devices, the RAID controller including a central processing unit and a consistency verification module, and the storage devices including a data storage module; A central processing unit, connected to at least two storage devices, is used to generate fast consistency check instructions and send the fast consistency check instructions to at least two storage devices; The storage device, connected to the central processing unit and consistency verification module, is used to obtain the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, calculate the data corresponding to the fast consistency check instruction, obtain the calculated value, and send the calculated value to the RAID controller. The consistency verification module is connected to the storage device and is used to compare the calculated values corresponding to the data at the same logical block address in at least two storage devices to obtain a third verification result.
[0011] In some embodiments, the central processing unit is further configured to obtain configuration information, generate a full consistency check instruction based on the configuration information, and obtain data returned by at least two storage devices based on the full consistency check instruction; The consistency verification module is also used to verify the data returned by at least two storage devices to obtain a third verification result.
[0012] In some embodiments, the consistency verification module includes a management control module, a configuration check module, and a data comparison module; The management and control module, connected to the central processing unit, the configuration check module, and the data comparison module, is used to generate configuration information; The configuration module is checked and connected to the management control module. It is used to configure the number of logical blocks for each verification based on the configuration information. The data comparison module, connected to the management and control module, is used to compare the calculated values corresponding to the data at the same logical block address in at least two storage devices when the number of logical blocks is greater than or equal to the preset number of logical blocks, or to verify the data corresponding to the same logical address in at least two storage devices to obtain a third verification result.
[0013] In some embodiments, the storage device includes a front-end module, a data processing module, an algorithm module, a back-end module, and a cache space; The front-end module is connected to the central processing unit, the consistency verification module, and the data processing module. It is used to receive fast consistency check instructions sent by the central processing unit, or to receive full consistency check instructions sent by the central processing unit. The fast consistency check instructions include a first logical address and a calculation algorithm type, while the full consistency check instructions include a second logical address. The data processing module, connected to the front-end module and the algorithm module, is used to calculate the data corresponding to the fast consistency check instruction based on the calculation algorithm type and obtain the calculated value. The algorithm module, connected to the data processing module and the backend module, is used to convert a first logical address into a first physical address, or a second logical address into a second physical address. The backend module, connected to the algorithm module and the data storage module, is used to retrieve the data corresponding to the fast consistency check instruction from the data storage module based on the first physical address, or to retrieve the data corresponding to the full consistency check instruction from the data storage module based on the second physical address, and cache the data corresponding to the fast consistency check instruction or the full consistency check instruction in the cache space.
[0014] In some embodiments, the data processing module includes a computing engine, which includes a management and control module, a computing configuration module, and a computing algorithm engine. The management and control module, connected to the front-end module, is used to parse the fast consistency check instructions to obtain the calculation algorithm type and the number of calculation logic blocks; The computation configuration module, connected to the management and control module and the algorithm engine, is used to configure the running parameters of the algorithm engine based on the computation algorithm type and the number of computation logic blocks. The calculation algorithm engine, connected to the management control module and the calculation configuration module, is used to calculate the data corresponding to the fast consistency check instruction based on the running parameters, obtain the calculated value, and send the calculated value to the front-end module.
[0015] The beneficial effects of the embodiments of this application are as follows: Unlike the prior art, the embodiments of this application provide a data verification method. This method obtains information about the storage device through the RAID controller, generates a fast consistency check instruction based on the information of the storage device, and the storage device obtains the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, calculates the data corresponding to the fast consistency check instruction, obtains the calculated value, and sends the calculated value to the RAID controller. The RAID controller compares the calculated values of at least two storage devices to obtain a first verification result. This can reduce the amount of data transmission between the RAID controller and multiple storage devices during the consistency check process and improve the efficiency of consistency checks on multiple storage devices. Attached Figure Description
[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0017] Figure 1 This is a schematic diagram illustrating an application environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a data verification method provided in an embodiment of this application; Figure 3 yes Figure 2 A detailed flowchart of step S202 in the process; Figure 4 This is an example schematic diagram of generating a calculated value based on CRC64 provided in the embodiments of this application; Figure 5 This is an example schematic diagram of generating a calculated value based on SHA256 provided in the embodiments of this application; Figure 6 yes Figure 2 A detailed flowchart of step S203 in the process; Figure 7 This is a schematic diagram of a process for performing a comprehensive consistency check on multiple storage devices, provided in an embodiment of this application. Figure 8 This is a flowchart illustrating the configuration of runtime parameters provided in an embodiment of this application; Figure 9 This is an example diagram illustrating the logical number of a single consistency check provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a data verification system provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a data verification system provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a consistency verification module provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a storage device provided in an embodiment of this application; Figure 14 This is a schematic diagram of the structure of a data processing module provided in an embodiment of this application; Figure 15 This is an example schematic diagram illustrating a storage device returning a calculated value, provided in an embodiment of this application. Figure 16 This is an example schematic diagram of data returned by a storage device according to an embodiment of this application.
[0018] Explanation of icon numbers: Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. In addition, the terms "first" and "second" used in this application do not limit the data, but only distinguish the same or similar items with basically the same function and effect.
[0021] The technical solution of this application is described in detail below with reference to the accompanying drawings: Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment provided in an embodiment of this application.
[0022] like Figure 1 As shown, the application environment 1000 includes a RAID controller 100 and at least two storage devices 200, with the RAID controller 100 communicatively connected to the at least two storage devices 200.
[0023] The RAID controller 100 is used to manage multiple storage devices to achieve redundant data storage and consistency assurance, and to promptly detect and correct data inconsistency issues, avoiding data loss or errors caused by storage device failures, transmission anomalies, etc.
[0024] Storage device 200 is used to store data.
[0025] In this embodiment, the RAID controller 100 periodically performs consistency checks on multiple storage devices to determine whether the data in the storage devices is consistent. In particular, for RAID1, which is a storage architecture based on data mirroring redundancy, RAID1 is used to manage at least two storage devices. One of the at least two storage devices is the source storage device, and the other storage devices are mirror storage devices of the source storage device. When writing data to the source storage device, the data is synchronously written to the mirror storage device. Therefore, it is necessary to periodically check whether there are differences in the data of each storage device to avoid data loss or errors when one of the storage devices fails.
[0026] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data verification method provided in an embodiment of this application.
[0027] This data verification method is applied to a data verification system.
[0028] The data verification system includes a RAID controller and at least two storage devices, with the RAID controller communicating with the at least two storage devices.
[0029] like Figure 2 As shown, the data verification method includes: Step S201: The RAID controller obtains information about the storage device and generates a fast consistency check instruction based on the information about the storage device.
[0030] In this embodiment of the application, the RAID controller periodically performs consistency checks on the data in at least two storage devices connected to the RAID controller to determine whether the data in the storage devices is consistent.
[0031] Specifically, before performing a consistency check on the data in at least two storage devices, an information retrieval request is sent to at least two storage devices connected to the RAID controller to obtain information about the storage devices, including the computing mode and the computing time required for the consistency check.
[0032] In the embodiments of this application, the calculation mode includes at least two data integrity verification algorithms. The types of data integrity verification algorithms include 64-bit Cyclic Redundancy Check 64 (CRC64), Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm (SHA), and National Secret Message Digest Algorithm 3 (SM3), wherein SHA includes SHA-1 and SHA-256.
[0033] In this embodiment, the data integrity verification algorithm is used to calculate the data on the storage device and generate a fixed-length message digest.
[0034] Specifically, after obtaining the storage device information, a fast consistency check instruction is generated based on the storage device information.
[0035] In this embodiment, the fast consistency check instruction is a vendor-defined command (VU), which is a dedicated communication command defined between the RAID controller and the storage device.
[0036] In this embodiment, the fast consistency check instruction includes a command code, a start logical block address, a number of logical blocks, a return data address, a calculation mode, and reserved bits. The calculation mode includes a calculation algorithm, which is one of at least two data integrity verification algorithms.
[0037] Step S202: The storage device obtains the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, calculates the data corresponding to the fast consistency check instruction, obtains the calculated value, and sends the calculated value to the RAID controller.
[0038] In this embodiment, the storage device includes a data storage module for storing data.
[0039] Specifically, when the storage device receives a fast consistency check command from the RAID controller, it reads data from the data storage module based on the start logical address and number of logical blocks in the fast consistency check command to obtain the data corresponding to the fast consistency check command. Based on the calculation algorithm in the fast consistency check command, it calculates the data corresponding to the fast consistency check command to obtain a message digest. The message digest is the calculated value, and the calculated value is sent to the RAID controller.
[0040] Please see Figure 3 , Figure 3 yes Figure 2 A detailed flowchart of step S202 in the process.
[0041] like Figure 3 As shown, step S202 includes: Step S221: The storage device initiates a fast consistency check based on the fast consistency check instruction.
[0042] Specifically, before retrieving the data corresponding to the fast consistency check instruction from the data storage module, the storage device initiates a fast consistency check based on the fast consistency check instruction.
[0043] Step S222: Calculate the data corresponding to the fast consistency check instruction based on the calculation algorithm to obtain the calculated value.
[0044] Specifically, the data corresponding to the fast consistency check instruction is calculated based on the computational algorithm to obtain a message digest, which is the calculated value.
[0045] In this embodiment of the application, the data length of the calculated value is determined according to the message digest algorithm.
[0046] Please see Figure 4 , Figure 4This is an example diagram illustrating a calculation value generated based on CRC64, provided in an embodiment of this application.
[0047] like Figure 4 As shown, the calculation mode in the fast consistency check instruction includes the CRC64 algorithm and the number of logical blocks. There are 256 logical blocks, each with a size of 512B. The total capacity of the original data is 256 * 512B = 128KB. After receiving the fast consistency check instruction, the storage device calculates the data corresponding to the 256 logical blocks based on the CRC64 algorithm to obtain the calculated value. The data length of the calculated value is 8B. The storage device sends the calculated value to the RAID controller.
[0048] Please see Figure 5 , Figure 5 This is an example schematic diagram of generating a calculated value based on SHA256 provided in the embodiments of this application.
[0049] like Figure 5 As shown, the calculation mode in the fast consistency check instruction includes the SHA256 algorithm and the number of logical blocks. The number of logical blocks is 2048, and the size of each logical block is 512B. The total capacity of the original data is 2048 * 512B = 1MB. After receiving the fast consistency check instruction, the storage device calculates the data corresponding to the 256 logical blocks based on the SHA256 algorithm to obtain the calculated value. The data length of the calculated value is 64B. The storage device sends the calculated value to the RAID controller.
[0050] In this embodiment, the size of the calculated value is much smaller than the size of the original data in the storage device. By directly sending the calculated value to the RAID controller, the amount of data transmission between the RAID controller and multiple storage devices can be reduced, thereby lowering the bandwidth utilization rate between the RAID controller and multiple storage devices.
[0051] Step S203: The RAID controller obtains the calculated values sent by at least two storage devices, compares the calculated values of at least two storage devices, and obtains the first verification result.
[0052] Specifically, the RAID controller obtains the calculated values sent by at least two storage devices, compares the calculated values of the at least two storage devices to determine whether the calculated values of the data corresponding to the same logical address in the at least two storage devices are the same, and obtains the first verification result.
[0053] In this embodiment of the application, the first verification result includes whether they are the same or different. If the first verification result is different, it is determined that the data corresponding to the same logical address in at least two storage devices are inconsistent. If the first verification result is the same, it is determined that the data corresponding to the same logical address in at least two storage devices are consistent.
[0054] In this embodiment of the application, when it is determined that the data corresponding to the same logical address in at least two storage devices is inconsistent, the storage device is marked with an anomaly information. The anomaly information includes the logical address corresponding to the inconsistent data and the storage device identifier. Data repair is triggered based on the marked anomaly information.
[0055] Please see Figure 6 , Figure 6 yes Figure 2 A detailed flowchart of step S203 is shown.
[0056] like Figure 6 As shown, step S203 includes: Step S231: Obtain the calculated values from at least two storage devices.
[0057] In this embodiment, the RAID controller includes a hardware accelerator and a preset comparison program.
[0058] Step S232: Determine whether the data length of the calculated value is less than the preset data length.
[0059] Specifically, the RAID controller determines whether the data length of the calculated value is less than the preset data length. If the data length of the calculated value is greater than or equal to the preset data length, it jumps to step S233. If the data length of the calculated value is less than the preset data length, it jumps to step S234.
[0060] Step S233: The RAID controller starts the hardware accelerator to compare the calculated values of at least two storage devices to obtain the first verification result.
[0061] Specifically, if the calculated data length is greater than or equal to the preset data length, the RAID controller activates the hardware accelerator to compare the calculated values of at least two storage devices and obtain the first verification result.
[0062] Step S234: The RAID controller compares the calculated values of at least two storage devices through a preset comparison program to obtain the first verification result.
[0063] Specifically, if the calculated data length is less than the preset data length, the RAID controller compares the calculated values of at least two storage devices using a preset comparison program to obtain the first verification result.
[0064] In this embodiment, the hardware accelerator is a hardware circuit in the RAID controller, which is used to process consistency verification of a large amount of data. The preset comparison program is a software logic program in the RAID controller, which is used to process consistency verification of a small amount of data.
[0065] Please see Figure 7 , Figure 7 This is a schematic diagram of a process for performing a comprehensive consistency check on multiple storage devices, provided in an embodiment of this application.
[0066] like Figure 7 As shown, the process for performing a comprehensive consistency check on multiple storage devices includes: Step S701: The RAID controller generates a full consistency check instruction based on a preset period and sends the full consistency check instruction to the storage devices to obtain the data corresponding to the full consistency check instruction returned by at least two storage devices.
[0067] In this embodiment, the RAID controller performs a full consistency check on the data of multiple storage devices at preset intervals. The preset interval is set according to specific circumstances, for example, the full consistency check is performed once a week.
[0068] Specifically, after the preset period is reached, the RAID controller generates a full consistency check command and sends the full consistency check command to the storage devices to obtain the data corresponding to the full consistency check command returned by at least two storage devices.
[0069] In this embodiment, the full consistency check instruction is used to instruct the storage device to return data. The full consistency check instruction is a VU command, which includes a command code, the address of the starting logical block, the number of logical blocks, the address of the returned data, and reserved bits.
[0070] In this embodiment, after receiving the full consistency check instruction, the storage device obtains the data corresponding to the full consistency check instruction from the data storage module based on the start logical block address and the number of logical blocks in the full consistency check instruction, and sends the data corresponding to the full consistency check instruction to the RAID controller.
[0071] Please see Figure 8 , Figure 8 This is a flowchart illustrating the configuration of runtime parameters provided in an embodiment of this application.
[0072] like Figure 8 As shown, the process for configuring runtime parameters includes: Step S801: Obtain configuration information.
[0073] In this embodiment of the application, the configuration information includes the number of logical blocks to be verified each time and the data integrity verification algorithm.
[0074] Step S802: Based on the configuration information, set the operating parameters of the hardware accelerator.
[0075] Specifically, based on the configuration information, the operating parameters of the hardware accelerator are set, including the number of logical blocks to be verified each time and the data integrity verification algorithm.
[0076] In this embodiment of the application, after the RAID controller receives the data corresponding to the full consistency check instruction returned by the storage device, it verifies the data corresponding to the full consistency check instruction returned by the storage device based on the operating parameters in the hardware accelerator.
[0077] Step S702: The RAID controller verifies the data corresponding to the full consistency check instructions returned by at least two storage devices to obtain a second verification result.
[0078] Specifically, after the RAID controller receives the data corresponding to the full consistency check instructions returned by at least two storage devices, it compares the data corresponding to the full consistency check instructions returned by at least two storage devices byte by byte to obtain the second verification result.
[0079] In this embodiment of the application, the second verification result includes whether they are the same or different. If the second verification result is different, it is determined that the data corresponding to the same logical address in at least two storage devices are inconsistent. If the second verification result is the same, it is determined that the data corresponding to the same logical address in at least two storage devices are consistent.
[0080] Please see Figure 9 , Figure 9 This is an example diagram illustrating the logical number of a single consistency check provided in the embodiments of this application.
[0081] like Figure 9 As shown, the total number of logical blocks in a single consistency check is divided into multiple logical block groups. A single consistency check within a cycle is divided into multiple consistency check instructions. Each logical block group corresponds to one consistency check instruction, and each logical block group includes multiple logical blocks. When performing a consistency check on the storage device, the storage device is checked based on multiple scan instructions until all consistency check instructions within a cycle are executed.
[0082] In this embodiment, fast consistency checks reduce the amount of data transfer between the storage device and the RAID controller, and reduce the computational load on the RAID controller. During the full consistency check, a hardware accelerator replaces the RAID controller's central processing unit to perform high-load comparisons, reducing the central processing unit's resource consumption, reducing system power consumption, and ensuring that the performance of normal RAID system services is not affected by the verification task.
[0083] Please see Figure 10 , Figure 10This is a schematic diagram of the structure of a data verification system provided in an embodiment of this application.
[0084] like Figure 10 As shown, the data verification system 1100 includes a RAID controller 100 and at least two storage devices 200. The RAID controller 100 includes a central processing unit 101 and a volatile storage module 102.
[0085] In this embodiment, the RAID controller 100 is used to manage multiple storage devices 200 to achieve redundant data storage and consistency assurance, and to promptly detect and correct data inconsistency problems, thereby avoiding data loss or errors caused by storage device 200 failures, transmission anomalies, etc.
[0086] The central processing unit 101 is connected to at least two storage devices 200. The central processing unit 101 is used to generate consistency verification instructions and send the consistency verification instructions to the at least two storage devices 200. The consistency verification instructions include logical address ranges.
[0087] Storage device 200 is connected to central processing unit 101. Storage device 200 is used to obtain the data corresponding to the logical address range based on the logical address range in the consistency check instruction, obtain the data corresponding to the consistency check instruction, and return the data corresponding to the consistency check instruction to RAID controller 100.
[0088] In this embodiment, the RAID controller 100 receives data corresponding to consistency verification instructions returned by at least two storage devices 200, caches the data in the volatile storage module 102, and the central processing unit 101 compares the data at the same logical address in the at least two storage devices 200 byte by byte to determine whether the data is consistent. If the data is consistent, the completion status of this verification is updated and the data in the volatile storage module 102 is released. If the data is inconsistent, a data repair process is triggered for the storage device 200 with the data error.
[0089] In this embodiment, after receiving the consistency verification command, the storage device 200 directly sends the original data stored in the storage device 200 to the RAID controller 100. The RAID controller 100 receives the original data sent by the storage device 200 and performs a comparison verification on the original data. For example, if there are two storage devices 200, and the data returned by the two storage devices 200 are original data 0 and original data 1 respectively, and the logical address corresponding to original data 0 is the same as the logical address corresponding to original data 1, the RAID controller 100 compares the original data 0 and original data 1 byte by byte to determine whether the original data 0 and original data 1 are consistent, so as to check whether the data of at least two storage devices 200 are consistent.
[0090] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a data verification system provided in an embodiment of this application.
[0091] like Figure 11 As shown, the data verification system 1100 includes a RAID controller 100 and at least two storage devices 200. The RAID controller 100 includes a central processing unit 101 and a consistency verification module 103, and the storage devices 200 include a data storage module 201.
[0092] The central processing unit 101 is connected to at least two storage devices 200. The central processing unit 101 is used to generate fast consistency check instructions and send the fast consistency check instructions to the at least two storage devices 200.
[0093] The storage device 200 is connected to the central processing unit 101 and the consistency verification module 103. The storage device 200 is used to obtain the data corresponding to the fast consistency check instruction from the data storage module 201 based on the fast consistency check instruction, calculate the data corresponding to the fast consistency check instruction, obtain the calculated value, and send the calculated value to the RAID controller 100.
[0094] The consistency verification module 103 is connected to the storage device 200. The consistency verification module 103 is used to compare the calculated values corresponding to the data of the same logical block address in at least two storage devices 200 to obtain a third verification result.
[0095] In this embodiment, the fast consistency check instruction is a vendor-defined command (VU), which is a dedicated communication command defined between the RAID controller 100 and the storage device 200.
[0096] In the embodiments of this application, the fast consistency check instruction includes a command code, the address of the starting logical block, the number of logical blocks, the return data address, the calculation mode, and reserved bits.
[0097] In this embodiment of the application, the fast consistency check instruction is used to instruct the storage device 200 to start a fast consistency check, so that the storage device 200 calculates the data corresponding to each logical block indicated in the fast consistency check instruction based on the calculation mode to obtain the calculated value.
[0098] In the embodiments of this application, the calculation mode includes calculation algorithm types, which include 64-bit Cyclic Redundancy Check 64 (CRC64), Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm (SHA), and National Secret Message Digest Algorithm 3 (SM3), wherein SHA includes SHA-1 and SHA-256.
[0099] In this embodiment, before the RAID controller 100 sends a fast consistency check instruction to the storage device 200, the RAID controller 100 sends an information acquisition instruction to the storage device 200 to obtain the types of computing algorithms supported by the storage device 200 and the computing time required for consistency checks. Based on the types of computing algorithms supported by the storage device 200, the RAID controller 100 sends a fast consistency check instruction to the storage device 200, so that the storage device 200 performs calculations on the data based on the calculation mode in the consistency check instruction to generate a message digest, i.e., a calculated value.
[0100] In this embodiment, the RAID controller 100 periodically performs consistency checks on multiple storage devices to determine whether the data in the storage devices is consistent. In particular, for RAID1, which is a storage architecture based on data mirroring redundancy, RAID1 is used to manage at least two storage devices. One of the at least two storage devices is the source storage device, and the other storage devices are mirror storage devices of the source storage device. When writing data to the source storage device, the data is synchronously written to the mirror storage device. Therefore, it is necessary to periodically check whether there are differences in the data of each storage device to avoid data loss or errors when one of the storage devices fails.
[0101] In this embodiment of the application, the third verification result includes being the same or different. If the third verification result is different, it is determined that the data corresponding to the same logical address in at least two storage devices 200 are inconsistent. If the third verification result is the same, it is determined that the data corresponding to the same logical address in at least two storage devices 200 are consistent.
[0102] In this embodiment, the central processing unit 101 is further configured to obtain configuration information, generate a full consistency check instruction based on the configuration information, and obtain data returned by at least two storage devices 200 based on the full consistency check instruction.
[0103] In this embodiment, the full consistency check instruction is used to instruct the storage device 200 to return data. The full consistency check instruction is a VU command, which includes a command code, the address of the starting logical block, the number of logical blocks, the address of the returned data, and reserved bits.
[0104] In this embodiment of the application, the consistency verification module 103 is also used to verify the data returned by at least two storage devices 200 to obtain a third verification result.
[0105] In this embodiment of the application, the third verification result includes being the same or different. If the third verification result is different, it is determined that the data corresponding to the same logical address in at least two storage devices 200 are inconsistent. If the verification result is the same, it is determined that the data corresponding to the same logical address in at least two storage devices 200 are consistent.
[0106] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a consistency verification module provided in an embodiment of this application.
[0107] like Figure 12 As shown, the consistency verification module 103 includes a management control module 131, a configuration check module 132, and a data comparison module 133.
[0108] The management control module 131 is connected to the central processing unit 101, the configuration check module 132, and the data comparison module 133. The management control module 131 is used to generate configuration information, which includes the number of logical blocks to be checked each time and the data integrity verification algorithm.
[0109] The configuration module 132 is connected to the management control module 131. The configuration module 132 is used to configure the number of logical blocks to be verified each time based on the configuration information.
[0110] The data comparison module 133 is connected to the management control module 131. The data comparison module 133 is used to compare the calculated values corresponding to the data of the same logical block address in at least two storage devices 200 when the number of logical blocks is greater than or equal to the preset number of logical blocks, or to verify the data corresponding to the same logical address in at least two storage devices 200 to obtain the verification result.
[0111] In this embodiment of the application, after the RAID controller 100 generates a fast consistency check instruction, it receives the calculated values returned by at least two storage devices 200, and the data comparison module 133 compares the calculated values corresponding to the data with the same logical block address in the at least two storage devices 200.
[0112] In this embodiment of the application, after the RAID controller 100 generates a full consistency check instruction, it receives data returned by at least two storage devices 200. The data comparison module 133 verifies the data corresponding to the same logical address in the at least two storage devices 200 and obtains the verification result.
[0113] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a storage device provided in an embodiment of this application.
[0114] like Figure 13 As shown, the storage device 200 includes a front-end module 202, a data processing module 203, an algorithm module 204, a back-end module 205, and a cache space.
[0115] The front-end module 202 is connected to the central processing unit 101, the consistency verification module 103, and the data processing module 203. The front-end module 202 is used to receive a fast consistency check instruction sent by the central processing unit 101, or to receive a full consistency check instruction sent by the central processing unit 101. The fast consistency check instruction includes a first logical address and a calculation algorithm type, and the full consistency check instruction includes a second logical address.
[0116] The data processing module 203 is connected to the front-end module 202 and the algorithm module 204. The data processing module 203 is used to calculate the data corresponding to the fast consistency check instruction based on the calculation algorithm type to obtain the calculated value.
[0117] The algorithm module 204 is connected to the data processing module 203 and the backend module 205. The algorithm module 204 is used to convert the first logical address into the first physical address, or to convert the second logical address into the second physical address.
[0118] The backend module 205 is connected to the algorithm module 204 and the data storage module 201. The backend module 205 is used to obtain the data corresponding to the fast consistency check instruction from the data storage module 201 based on the first physical address, or to obtain the data corresponding to the full consistency check instruction from the data storage module 201 based on the second physical address, and cache the data corresponding to the fast consistency check instruction or the data corresponding to the full consistency check instruction in the cache space.
[0119] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a data processing module provided in an embodiment of this application.
[0120] like Figure 14As shown, the data processing module 203 includes a computing engine 231, which includes a management and control module 2311, a computing configuration module 2312, and a computing algorithm engine 2313.
[0121] The management control module 2311 is connected to the front-end module 202. The management control module 2311 is used to parse the fast consistency check instructions to obtain the calculation algorithm type and the number of calculation logic blocks.
[0122] The calculation configuration module 2312 is connected to the management and control module 2311 and the algorithm engine. The calculation configuration module 2312 is used to configure the running parameters of the algorithm engine based on the calculation algorithm type and the number of calculation logic blocks. The running parameters include the number of logic blocks to be verified each time and the data integrity verification algorithm.
[0123] The calculation algorithm engine 2313 is connected to the management and control module 2311 and the calculation configuration module 2312. The calculation algorithm engine 2313 is used to calculate the data corresponding to the fast consistency check instruction based on the data integrity verification algorithm in the running parameters, obtain the calculated value, and send the calculated value to the front-end module 202.
[0124] Please see Figure 15 , Figure 15 This is an example schematic diagram illustrating the return of calculated values by a storage device according to an embodiment of this application.
[0125] like Figure 15 As shown, the RAID controller 100 sends fast consistency check commands to multiple storage devices 200. After receiving the fast consistency check command, the computing engine 231 in the storage device 200 calculates the data corresponding to the fast consistency check command, obtains the calculated value, and returns the calculated value to the RAID controller 100.
[0126] The RAID controller 100 receives calculated values returned by multiple storage devices 200 and compares whether the calculated values corresponding to the same logical block in different storage devices are the same. If they are the same, it is determined that the data corresponding to the same logical address in at least two storage devices is consistent. If they are not the same, it is determined that the data corresponding to the same logical address in at least two storage devices is inconsistent, and then the storage devices are troubleshooted and maintained.
[0127] Please see Figure 16 , Figure 16 This is an example schematic diagram of data returned by a storage device according to an embodiment of this application.
[0128] like Figure 16As shown, the RAID controller 100 sends a full consistency check command to multiple storage devices 200. After receiving the full consistency check command, the storage device 200 sends the data corresponding to the full consistency check command to the RAID controller 100. The hardware accelerator in the RAID controller 100 compares the original data corresponding to the same logical block in different storage devices byte by byte to determine whether the original data corresponding to the same logical block in different storage devices is the same, so as to check whether the original data corresponding to the same logical block in at least two storage devices is consistent.
[0129] In this embodiment, the amount of data transfer between the storage device 200 and the RAID controller 100 is reduced by fast consistency checks, and the amount of computation of the RAID controller 100 is also reduced. During the full consistency check, a hardware accelerator replaces the central processing unit 101 of the RAID controller 100 to perform high-load comparisons, which reduces the resource consumption of the central processing unit 101, reduces the power consumption of the system, and ensures that the performance of normal business operations of the RAID system is not affected by the verification task.
[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The non-volatile computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations as described above in different aspects of this application, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A data verification method, characterized in that, The method, applied to a data verification system, includes a RAID controller and at least two storage devices communicatively connected to the RAID controller, each storage device including a data storage module, and comprises: The RAID controller obtains information about the storage device and generates a fast consistency check instruction based on the information about the storage device. The storage device obtains the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, performs calculations on the data corresponding to the fast consistency check instruction to obtain a calculated value, and sends the calculated value to the RAID controller. The RAID controller acquires the calculated values sent by at least two of the storage devices, compares the calculated values of the at least two storage devices, and obtains a first verification result.
2. The method according to claim 1, characterized in that, The information of the storage device includes a computing mode, the computing mode includes at least two data integrity verification algorithms, and the fast consistency check instruction includes a computing algorithm, the computing algorithm being one of the at least two data integrity verification algorithms. Before obtaining the data corresponding to the fast consistency check instruction from the data storage module, the method further includes: The storage device initiates a fast consistency check based on the fast consistency check instruction; The calculation of the data corresponding to the fast consistency check instruction to obtain the calculated value includes: The calculation algorithm is used to calculate the data corresponding to the fast consistency check instruction to obtain the calculated value.
3. The method according to claim 1, characterized in that, The RAID controller includes a hardware accelerator and a preset comparison program; The step of comparing the calculated values of at least two of the storage devices to obtain a first verification result further includes: If the data length of the calculated value is greater than or equal to the preset data length, the RAID controller activates the hardware accelerator to compare the calculated values of at least two storage devices to obtain the first verification result. If the data length of the calculated value is less than the preset data length, the RAID controller compares the calculated values of at least two storage devices through the preset comparison program to obtain the first verification result.
4. The method according to claim 1, characterized in that, The method further includes: The RAID controller generates a full consistency check instruction based on a preset period, and sends the full consistency check instruction to the storage device to obtain data corresponding to the full consistency check instruction returned by at least two storage devices. The RAID controller verifies the data corresponding to the full consistency check instructions returned by at least two storage devices to obtain a second verification result.
5. The method according to claim 4, characterized in that, Before generating the full consistency check instruction, the method further includes: Get configuration information; Based on the configuration information, the operating parameters of the hardware accelerator are set; After obtaining the data corresponding to the full consistency check instructions returned by at least two storage devices, the method further includes: Based on the operating parameters of the hardware accelerator, the data corresponding to the full consistency check instructions returned by at least two storage devices are verified to obtain the second verification result.
6. A data verification system, characterized in that, The system includes a RAID controller and at least two storage devices. The RAID controller includes a central processing unit and a consistency verification module. The storage devices include a data storage module. The central processing unit is connected to the at least two storage devices and is used to generate fast consistency check instructions and send the fast consistency check instructions to the at least two storage devices. The storage device is connected to the central processing unit and the consistency verification module. It is used to obtain the data corresponding to the fast consistency check instruction from the data storage module based on the fast consistency check instruction, calculate the data corresponding to the fast consistency check instruction, obtain the calculated value, and send the calculated value to the RAID controller. The consistency verification module is connected to the storage device and is used to compare the calculated values corresponding to the data of the same logical block address in at least two of the storage devices to obtain a third verification result.
7. The system according to claim 6, characterized in that, The central processing unit is also used to obtain configuration information, generate a full consistency check instruction based on the configuration information, and obtain data returned by at least two of the storage devices based on the full consistency check instruction. The consistency verification module is also used to verify the data returned by the at least two storage devices to obtain a third verification result.
8. The system according to claim 7, characterized in that, The consistency verification module includes a management control module, a configuration check module, and a data comparison module; The management and control module is connected to the central processing unit, the configuration check module, and the data comparison module, and is used to generate configuration information. The inspection configuration module is connected to the management control module and is used to configure the number of logical blocks to be checked each time based on the configuration information. The data comparison module, connected to the management and control module, is used to compare the calculated values corresponding to the data at the same logical block address in the at least two storage devices when the number of logical blocks is greater than or equal to the preset number of logical blocks, or to verify the data corresponding to the same logical address in the at least two storage devices to obtain the third verification result.
9. The system according to claim 6, characterized in that, The storage device includes a front-end module, a data processing module, an algorithm module, a back-end module, and a cache space; The front-end module is connected to the central processing unit, the consistency verification module, and the data processing module, and is used to receive a fast consistency check instruction sent by the central processing unit, or to receive a full consistency check instruction sent by the central processing unit. The fast consistency check instruction includes a first logical address and a calculation algorithm type, and the full consistency check instruction includes a second logical address. The data processing module is connected to the front-end module and the algorithm module, and is used to calculate the data corresponding to the fast consistency check instruction based on the calculation algorithm type to obtain the calculated value; The algorithm module is connected to the data processing module and the backend module, and is used to convert the first logical address into the first physical address, or to convert the second logical address into the second physical address. The backend module is connected to the algorithm module and the data storage module, and is used to obtain the data corresponding to the fast consistency check instruction from the data storage module based on the first physical address, or to obtain the data corresponding to the full consistency check instruction from the data storage module based on the second physical address, and cache the data corresponding to the fast consistency check instruction or the data corresponding to the full consistency check instruction in the cache space.
10. The system according to claim 6, characterized in that, The data processing module includes a computing engine, which includes a management and control module, a computing configuration module, and a computing algorithm engine. The management and control module is connected to the front-end module and is used to parse the fast consistency check instruction to obtain the calculation algorithm type and the number of calculation logic blocks. The computing configuration module is connected to the management control module and the algorithm engine, and is used to configure the running parameters of the algorithm engine based on the computing algorithm type and the number of computing logic blocks. The calculation algorithm engine is connected to the management control module and the calculation configuration module. It is used to calculate the data corresponding to the fast consistency check instruction based on the running parameters, obtain the calculated value, and send the calculated value to the front-end module.