Data consistency verification method and device for storage object, equipment and medium

Abstract

The invention provides a data consistency verification method and device for a storage object, equipment and a medium. The method comprises the steps of obtaining template data with known data size and content; circularly writing the template data in a to-be-tested area of the storage object; in the process of cyclically writing the template data, simulating and executing the fault and recovery operation of the storage object; after the fault of the storage object is recovered, reading write-in data in the to-be-tested area; and verifying the data consistency of the to-be-tested area according to a comparison result of the write-in data and the template data. According to the technical scheme, the problem that the data consistency of the last write-in storage object when the fault occurs cannot be verified in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of storage, and particularly relates to a data consistency verification method and device for a storage object, an equipment and a medium. BACKGROUND

[0002] In the field of storage, data consistency testing as a key test item is widely used in products such as hard disks, disk array cards and file systems.

[0003] Data inconsistency may cause business logic errors (such as bank transfer amount mismatch) or system crashes (such as data conflict between distributed database nodes). Through static testing (such as code review) and dynamic testing (such as stress testing and fault injection), defects in concurrent control or problems in redundant data synchronization can be found in advance to avoid production environment accidents. In storage testing, how to construct a scenario to trigger potential data consistency problems and prevent the impact of data inconsistency in advance has become an important work in the storage industry testing. SUMMARY

[0004] The present application aims to provide a data consistency verification method and device for a storage object, an equipment and a medium, and aims to solve the problem that related technologies cannot verify the consistency of a piece of data last written to a storage object after a device fails and recovers.

[0005] According to a first aspect of the present application, a data consistency verification method for a storage object is provided, comprising:

[0006] Obtaining template data with known data size and content;

[0007] Cyclically writing the template data to a to-be-tested area of the storage object;

[0008] Simulating a failure and recovery operation of the storage object during the cyclic writing of the template data;

[0009] Reading out the written data in the to-be-tested area after the failure and recovery of the storage object;

[0010] Verifying the data consistency of the to-be-tested area according to the comparison result of the written data and the template data.

[0011] The application uses template data known in advance to replace random data generated in real time from a test tool in the related art, so that each fixed offset position of a to-be-tested region covered by the template data corresponds to the data written, and even if the to-be-tested region is written again by the template data, the data corresponding to the offset position will not change. Therefore, after the simulated fault operation is restored, the data written in the to-be-tested region can be read out and compared with the template data to verify whether the data at each offset position meets the expectation, thereby overcoming the problem that the related art cannot verify the data consistency of the last write to the storage object when a fault occurs.

[0012] In an optional embodiment, the data size of the template data is smaller than the storage space of the to-be-tested region.

[0013] In an optional embodiment, before the template data is cyclically written to the to-be-tested region of the storage object, the method further comprises:

[0014] The template data is written to the to-be-tested region multiple times to completely cover the to-be-tested region.

[0015] In an optional embodiment, according to the comparison result of the written data and the template data, the data consistency of the to-be-tested region is verified, comprising:

[0016] The offset position of the to-be-tested region and the corresponding relationship between the data in the template data are determined by using the data size of the template data and the starting address of the to-be-tested region;

[0017] The to-be-compared data in the template data corresponding to the read position of the written data is determined based on the corresponding relationship;

[0018] The data consistency is verified by comparing the written data and the to-be-compared data.

[0019] According to a second aspect of the application, a data consistency verification device of a storage object is provided, comprising:

[0020] An acquisition module is configured to acquire template data with a known data size and content;

[0021] A cyclic writing module is configured to cyclically write the template data to a to-be-tested region of the storage object;

[0022] A fault and recovery module is configured to simulate the execution of a fault and recovery operation of the storage object during the cyclic writing of the template data;

[0023] A data reading module is configured to read the written data in the to-be-tested region after the fault and recovery of the storage object;

[0024] A verification module configured to verify data consistency of the to-be-tested region according to a comparison result of the write data and the template data.

[0025] In an optional implementation, the template data has a data size smaller than a storage space of the to-be-tested region.

[0026] In an optional implementation, before the cyclic write module, the apparatus further comprises:

[0027] A data write module configured to write the template data to the to-be-tested region for multiple times to completely cover the to-be-tested region.

[0028] In an optional implementation, the verification module comprises:

[0029] A first determination submodule configured to determine a correspondence between an offset position of the to-be-tested region and data in the template data by using a data size of the template data and a start address of the to-be-tested region;

[0030] A second determination submodule configured to determine, based on the correspondence, to-be-compared data in the template data corresponding to a readout position of the write data;

[0031] A comparison submodule configured to verify the data consistency by comparing the write data and the to-be-compared data.

[0032] The third aspect of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method of the first aspect.

[0033] The fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a plurality of instructions, and the plurality of instructions can be read and executed by a processor to execute the method of the first aspect.

[0034] Other features and advantages of the present application will be described in the following description and become apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application can be achieved and obtained by the structure and flow illustrated in the description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0036] Figure 1 is a flowchart of a data consistency verification method of a storage object according to an example embodiment of the present application.

[0037] Figure 2 is a flowchart of a data consistency verification method of a storage object according to an example embodiment of the present application.

[0038] Figure 3 is a flowchart of a data consistency verification method of a storage object according to an example embodiment of the present application.

[0039] Figure 4 is a flowchart of a data consistency verification method of a storage object according to an example embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] The method provided by the present application can be implemented in a terminal environment, which can include one or more of the following components: a processor, a memory and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the embodiments described below.

[0042] The processor can include one or more processing cores. The processor connects various parts in the entire terminal through various interfaces and lines, and performs various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0043] The memory can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory can be used to store instructions, programs, codes, code sets or instructions.

[0044] The display screen is used to display the user interface of each application program.

[0045] In addition, those skilled in the art can understand that the structure of the terminal described above does not constitute a limitation on the terminal, and the terminal can include more or fewer components, or combine certain components, or different component arrangements. For example, the terminal also includes radio frequency circuitry, input units, sensors, audio circuitry, power supplies, and the like, which are not described here.

[0046] The concept of data inconsistency is explained below.

[0047] 1. Data inconsistency: the actual data written does not match the expected data written.

[0048] 2. Problems caused by data inconsistency: may cause user data loss, file system structure damage, business exceptions, system downtime, etc.

[0049] 3. Test design difficulties: the test method of related technology cannot detect data inconsistency that occurs during user common operations such as reboot, unexpected power failure, or hard disk removal, due to its fixed logic design. The test method of related technology is to write a data block and then read it immediately to verify, but after the above user operations, the system or device may no longer work, so it is not possible to verify the data consistency of the last write data block before the system or device "not working" state.

[0050] 4. Possible problems with storage objects: The sector size of non-volatile storage media is usually 512 bytes or 4k bytes, such as a 4k sector size hard disk. Each I / O must be written to non-volatile storage media as complete 4k data. If this I / O is only 3k, the remaining 1k will be padded, such as padding to 0. If a system reboot, power reboot, etc. occurs, it also needs to be padded. If an accident occurs at the position where 3k data is written, the remaining 1k will appear data inconsistency if padded to 0 or other data.

[0051] The data consistency verification scheme in related technology is as follows:

[0052] Configure the data verification parameters of the I / O test tool (such as fio, vdbench), such as the command:

[0053] fio-filename= / dev / sdb-direct=1-ioengine=libaio-exitall_on_error=1-bs=4k-rw=write-iodepth=32-numjobs=1-do_verify=1-verify=md5-verify_backlog=1-verify_dump=1--group_reporting-runtime=600-name=test

[0054] The scheme reads out the data immediately after writing the data to the target position, compares the data with the expected data, and determines whether there is a consistency problem.

[0055] In the related art scheme, if a shutdown operation is performed when data is written, the last I / O operation at the time of shutdown cannot be read out immediately by the test tool, so the consistency of the data cannot be verified, and data inconsistency may occur in this scenario.

[0056] Based on the above analysis, as shown in Figure 1 The present application exemplarily proposes a data consistency verification method of a storage object, comprising:

[0057] Step S101: Obtain template data with known data size and content;

[0058] Step S102: Write the template data to the test area of the storage object in a loop;

[0059] Step S103: Simulate the failure and recovery operations of the storage object during the loop writing of the template data;

[0060] Step S104: Read out the written data in the test area after the failure recovery of the storage object;

[0061] Step S105: Verify the data consistency of the test area according to the comparison result of the written data and the template data.

[0062] Exemplarily, the storage object can be a hardware storage module or a file storage system capable of reading and writing data. The template data can be a file constructed by random numbers generated by a corresponding command such as / dev / urandom in the test preparation stage, such as a 5MB data file with the following content: 5a3fe72c910db844...7b12e45a8d0f3e21. It can be understood that the template data can also be a data file with known data size and content generated by other means in advance. Compared with the related art, the present application adopts two-stage verification by two I / O operations. The first I / O operation only writes data, that is, the template data is written to the test area of the storage object in a loop, and the failure operation such as reboot and power-off is simulated during the loop writing, and then the failure is recovered, such as reboot completion, power-on after power-off and startup, the written data is read from the test area again by I / O operation, and the read written data is verified for data consistency by using the template data.

[0063] Exemplarily, as shown in Figure 2As shown, assuming that the storage object to be tested is a disk, data consistency of a specified region A of the disk is checked, and template data is cyclically written in the specified region A. Meanwhile, a fault simulation operation such as restarting (or power-off or hard disk dismounting) of a server where the disk is located is performed during the writing process. After the fault simulation operation is restored, whether there is a data inconsistency problem is determined by comparing the written data in the specified region A with the template data. In this way, whether the last I / O at the instant of shutdown meets the expectation can be verified.

[0064] Exemplarily, through practice, the present application can be applied to verify data consistency problems of products such as hard disks of different brands, disk array cards, SAS expansion backplanes, and also supports verification of data consistency of products such as memories and file systems. In addition, the scheme of the present application is applicable to various operating systems such as Linux, Windows, and MacOS. It should be noted that different operating systems can use different commands to execute the steps of the above scheme, and of course, the commands in the above example can be replaced by corresponding tools, such as replacing the fio command with the dd tool. After the fault is restored, other steps can be implemented using scripts such as shell or Powershell. It can be understood that all the steps in the above entire scheme can also be implemented using scripts such as shell or Powershell.

[0065] The present application uses template data known as content, instead of random data generated in real time from a test tool in the related art, so that the data written in each fixed offset position of the test region cyclically covered by the template data is known, and even if the test region is written again by the template data, the expected data corresponding to the offset position will not change. Therefore, after the simulated fault operation is restored, the data written in the test region can be read out and compared with the template data to verify whether the data at each offset position meets the expectation, and whether the last write data after the fault occurs is unexpectedly written with unexpected data, thereby overcoming the problem that the related art cannot verify the data consistency of the last write storage object when the fault occurs.

[0066] In some optional implementations, the data size of the template data is smaller than the storage space of the test region.

[0067] Exemplarily, the data size of the template data can be much smaller than the test region, for example, the data size of the template data is in the order of MB, and the test region is in the order of GB, so that cyclically writing multiple times can completely cover the test region.

[0068] Since the data size of the template data is small, the template data can be easily managed, and the computational complexity can be reduced during data consistency verification.

[0069] In some optional implementations, before cyclically writing the template data to the area to be tested of the storage object, the method further includes:

[0070] Write template data to the area to be tested multiple times to completely cover the area to be tested.

[0071] For example, Figure 3 As shown, a 5MB template data can be constructed using a command such as "dd if= / dev / urandom of=temp.bin bs=5M count=1". The template data is written sequentially and cyclically into a specified location range of the test area of ​​the storage object to be tested, such as the drive letter / dev / sdb, and the total data size is specified, such as 10GB. This ensures that the test area has been overwritten by the template data once, so that after simulating a fault operation and recovering, the corresponding data in the template data is stored at each fixed offset position in the test area. For example, the following command can be used to ensure that the test area is completely covered by the template data: dd if(input file)=temp.bin of(output file)= / dev / sdb bs(block size)=5M count=2048seek=0.

[0072] Continue writing the template data to the specified location range in the test area using a loop write command such as fio. Once 10GB is written, writing automatically restarts from the beginning. During this process, I / O continues writing, looping back and forth within the 10GB space. During this I / O process, simulate power or storage failures, such as shutdown, restart, power reset, unexpected power outage, and hard drive removal. For example, you can use the trigger function of the fio loop write command to simulate these failures.

[0073] Restore the above operations to normal, such as booting the system to restore the system, reinserting the hard drive, etc. Use the loop write command fio to read the 10GB data in the above test area and compare it with the 5MB template data. For example, you can use the hash value to verify whether the 10GB data is completely overwritten by the 5MB template data.

[0074] This approach can ensure that the area to be tested is completely covered by the template data during the fault simulation operation, thereby ensuring that the expected data read from the area to be tested is known when verifying consistency.

[0075] In some optional implementations, the step of verifying data consistency of the area to be tested based on a comparison result between the written data and the template data includes:

[0076] The data size of the template data and the start address of the to-be-tested region are used to determine the correspondence between the offset position of the to-be-tested region and the data in the template data.

[0077] Based on the correspondence, the to-be-compared data in the template data corresponding to the readout position of the written data is determined.

[0078] The data consistency is verified by comparing the written data and the to-be-compared data.

[0079] Exemplarily, since the data size of the template data is much smaller than the to-be-tested region, and before the fault simulation operation, the template data is written to the to-be-tested region for multiple times, so that the template data completely covers the to-be-tested region once. Therefore, after the fault recovery, the to-be-tested region is also filled with the data in the template data, and the written data at each offset position of the to-be-tested region can also be determined according to the template data. For example, the 5MB template data is written from the start position of the to-be-tested region, that is, the first data block with a size of 5MB starting from the offset position 0 should be a template data, the second data block with a size of 5MB starting from the offset position 5MB is also a template data, and so on, so that the correspondence between each offset position of the to-be-tested region and the data in the template data can be determined. In the verification, the written data is read out from the to-be-tested region, and whether the data in the template data corresponding to the readout position of the written data (that is, the to-be-compared data) is consistent with the written data is determined according to the readout position of the written data and the above correspondence. If it is consistent, it means that the data consistency verification is passed, and if it is not consistent, it means that the data consistency verification is not passed, and the storage object has a data consistency problem.

[0080] In this way, since the data at each fixed offset position of the to-be-tested region has a correspondence with the data in the data template, it can be determined in advance that the written data read out from the to-be-tested region should be what kind of data, so that the data consistency problem of the storage product can be compared and verified.

[0081] Correspondingly, as shown in Figure 4 Exemplarily, the present application provides a data consistency verification device of a storage object, which comprises:

[0082] The acquisition module 401 is configured to acquire template data with known data size and content.

[0083] The cyclic writing module 402 is configured to cyclically write the template data to the to-be-tested region of the storage object.

[0084] The fault and recovery module 403 is configured to simulate the execution of the fault and recovery operation of the storage object in the process of cyclically writing the template data.

[0085] The data reading module 404 is configured to read out the written data in the to-be-tested region after the fault recovery of the storage object.

[0086] The verification module 405 is configured to verify the data consistency of the to-be-tested region according to the comparison result of the write data and the template data.

[0087] In some optional implementations, the data size of the template data is smaller than the storage space of the to-be-tested region.

[0088] In some optional implementations, before the cyclic writing module, the device further comprises:

[0089] The data writing module is configured to write the template data into the to-be-tested region for multiple times to completely cover the to-be-tested region.

[0090] In some optional implementations, the verification module comprises:

[0091] The first determination submodule is configured to determine the corresponding relationship between the offset position of the to-be-tested region and the data in the template data by using the data size of the template data and the start address of the to-be-tested region.

[0092] The second determination submodule is configured to determine the to-be-compared data in the template data corresponding to the read-out position of the write data based on the corresponding relationship.

[0093] The comparison submodule is configured to verify the data consistency by comparing the write data and the to-be-compared data.

[0094] The device corresponds to the above-mentioned data consistency verification method of the storage object, and specific details can be referred to the description of the data consistency verification method of the storage object in the above text, which will not be described here.

[0095] It can be understood that the circuit structure, name and parameter described in the above embodiments are only examples. Those skilled in the art can also combine and adjust the structural features of the above multiple embodiments according to the use needs, and the concept of the present application should not be limited to the specific details of the above examples.

[0096] The application further provides an electronic device including a processor and a memory. The memory stores a plurality of instructions, and the processor is configured to read the instructions and perform any of the aforementioned data consistency verification methods. The processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0097] The memory is a non-transitory computer-readable storage medium and can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules corresponding to the methods in the embodiments of the application. The processor performs various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the methods in the method embodiments.

[0098] The memory can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] Although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for verifying data consistency of a storage object, characterized in that: include: Get template data with known data size and content; cyclically writing the template data to the area to be tested of the storage object; During the process of cyclically writing the template data, simulating and executing the failure and recovery operations of the storage object; After the storage object recovers from the fault, reading out the written data in the area to be tested; The data consistency of the area to be tested is verified according to a comparison result between the written data and the template data.

2. The method for verifying data consistency of a storage object according to claim 1, wherein: The data size of the template data is smaller than the storage space of the area to be tested.

3. The method for verifying data consistency of a storage object according to claim 2, wherein: Before cyclically writing the template data to the area to be tested of the storage object, the method further includes: The template data is written into the area to be tested multiple times to completely cover the area to be tested.

4. The method for verifying data consistency of a storage object according to any one of claims 1 to 3, characterized in that: The verifying the data consistency of the area to be tested according to the comparison result between the written data and the template data includes: Determine the correspondence between the offset position of the area to be tested and the data in the template data by using the data size of the template data and the starting address of the area to be tested; Determine, based on the corresponding relationship, the data to be compared in the template data corresponding to the read position of the written data; The data consistency is verified by comparing the written data with the data to be compared.

5. A data consistency verification device for a storage object, characterized in that: include: The acquisition module is used to obtain template data with known data size and content; a cyclic writing module, configured to cyclically write the template data to the area to be tested of the storage object; A fault and recovery module, configured to simulate and execute fault and recovery operations of the storage object during the process of cyclically writing the template data; A data reading module, configured to read out the written data in the area to be tested after the storage object recovers from a fault; The verification module is used to verify the data consistency of the area to be tested based on the comparison result between the written data and the template data.

6. The data consistency verification device for a storage object according to claim 5, characterized in that: The data size of the template data is smaller than the storage space of the area to be tested.

7. The data consistency verification device for a storage object according to claim 6, characterized in that: Before the cyclic writing module, the device further includes: The data writing module is used to write the template data into the area to be tested multiple times to completely cover the area to be tested.

8. The data consistency verification device for a storage object according to any one of claims 5 to 7, characterized in that: The verification module includes: A first determining submodule is configured to determine a correspondence between an offset position of the area to be tested and data in the template data by using the data size of the template data and the starting address of the area to be tested; A second determining submodule is configured to determine, based on the corresponding relationship, the data to be compared in the template data corresponding to the read position of the written data; The comparison submodule is configured to verify the data consistency by comparing the written data with the data to be compared.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the data consistency verification method of the storage object according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed by the method for verifying data consistency of a storage object according to any one of claims 1 to 4.

Images