Snapshot verification method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a snapshot verification method and device, electronic equipment and a readable storage medium, and relates to the technical field of snapshot verification, and the method can determine a target snapshot type of backup data and a target writing mode of source data under the condition of performing backup based on a snapshot. In order to flexibly cope with snapshot consistency verification in different storage scenes, different snapshot types and verification modes corresponding to different write-in modes are preset for decision setting, and the decision setting is recorded in decision information. And after the target snapshot type and the target write-in mode are obtained, the decision information can be obtained, so that the current adaptive target verification mode is determined and executed. The method has the technical effects that the verification strategy is automatically selected according to different writing modes, snapshot types and the like, so that the environmental adaptability can be improved; in combination with a plurality of verification modes, a test can be covered from multiple dimensions, and a blind area of data verification is eliminated.

Description

Technical Field

[0001] The present application relates to the field of storage technologies, and in particular, to a snapshot verification method, apparatus, electronic device, and readable storage medium. Background Art

[0002] Snapshot technology refers to the replication of a certain data set at a specific moment. Currently, snapshot is mostly used for data protection in storage devices. Traditional snapshots are divided into COW (Copy-On-Write) snapshots and ROW (Redirect-On-Write) snapshots; currently, there is also a high-density snapshot function technology, namely Continuous Data Protection (CDP) technology. In CDP technology, every time there is a data change in the source volume, a mapping table will be generated to record the timestamp and the LBA (Logical Block Address) of the new data. When recovering data, the LBA corresponding to the time point in the mapping table is searched to read the data at that time point.

[0003] The main basis for realizing data protection based on snapshots is that based on snapshot technology, the stored backup data can be used to roll back the source data and recover the data. Therefore, it is crucial to ensure that the snapshot data created each time is correct, that is, the backup data of the snapshot is consistent with the source data of the source volume.

[0004] The currently adopted verification method extracts the snapshot data and then performs hashing, and compares it with the hash value of the data on the actual source volume. This verification method has certain limitations. It can only perform file verification of the file system and cannot verify block data; moreover, for large amounts of data and random data, there is a lack of effective verification means.

[0005] Therefore, for those skilled in the art, how to flexibly perform consistency verification on snapshots is a technical problem that needs to be solved urgently at present. Summary of the Invention

[0006] The present application provides a snapshot verification method, apparatus, electronic device, and readable storage medium, which can flexibly handle the storage environment and select an appropriate verification method to perform consistency verification on snapshots.

[0007] A snapshot verification method includes:

[0008] When backing up source data based on a snapshot, determining the target snapshot type of the backup data and the target writing method of the source data;

[0009] Obtaining decision information recording the snapshot type, writing method, and verification method;

[0010] Determine a target verification method that matches both the target snapshot type and the target writing method using decision-making information;

[0011] Execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

[0012] A snapshot verification device, comprising:

[0013] A snapshot parameter determination module, configured to determine the target snapshot type of the backup data and the target writing method of the source data when backing up the source data based on a snapshot;

[0014] A decision-making information acquisition module, configured to acquire decision-making information recording the snapshot type, the writing method, and the verification method;

[0015] A verification method determination module, configured to determine a target verification method that matches both the target snapshot type and the target writing method using decision-making information;

[0016] A verification execution module, configured to execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

[0017] This application also provides an electronic device, comprising: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above snapshot verification methods when executing the computer program.

[0018] This application also provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any one of the above snapshot verification methods when executed by a processor.

[0019] This application also provides a computer program product, comprising a computer program, and the computer program implements the steps of any one of the above snapshot verification methods when executed by a processor.

[0020] In this application, when backing up the source data based on a snapshot, first determine the target snapshot type of the backup data and the target writing method of the source data. For different snapshot types and different writing methods of metadata, to achieve consistency verification between the source data and the backup data, the corresponding verification method needs to be changed. In this application, in order to flexibly handle snapshot consistency verification in different storage scenarios, the verification methods corresponding to different snapshot types and different writing methods can be preset for decision-making settings and recorded in the decision-making information. After obtaining the target snapshot type and the target writing method, the decision-making information can be obtained, and thus the currently adapted target verification method can be determined. Then, execute the verification process corresponding to the target verification method, and consistency verification can be performed on the backup data and the source data.

[0021] Technical effects of the present application: Through automated integration, it is possible to automatically select verification strategies according to different writing modes, snapshot types, etc., which can solve the problem of insufficient environmental adaptability during data verification; by combining multiple verification methods, it is possible to cover tests from multiple dimensions and eliminate blind spots in data verification, so as to quickly locate the root cause when data inconsistency occurs. Brief Description of the Drawings

[0022] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0023] Figure 1 Flow chart of a snapshot verification method provided by an embodiment of the present application;

[0024] Figure 2 Implementation flowchart of a block verification method provided by an embodiment of the present application;

[0025] Figure 3 Implementation flowchart of a hierarchical verification method provided by an embodiment of the present application;

[0026] Figure 4 Implementation flowchart of a data simulation verification method provided by an embodiment of the present application;

[0027] Figure 5 Structural schematic diagram of a snapshot verification device provided by an embodiment of the present application;

[0028] Figure 6 Structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0029] Figure 7 Specific structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0031] It should be noted that in the description of this application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0032] To enable those skilled in the art of this technology to better understand the solution of this application, the following further details this application in conjunction with the accompanying drawings and specific embodiments.

[0033] An embodiment of this application provides a snapshot verification method, and the method is described in detail in combination with the execution process of the snapshot verification method.

[0034] Please refer to Figure 1 , Figure 1 which is the implementation flowchart of a snapshot verification method provided by an embodiment of this application. The method includes:

[0035] S101. When backing up source data based on a snapshot, determine the target snapshot type of the backup data and the target writing method of the source data.

[0036] In this embodiment, in order to adapt to the verification method corresponding to the snapshot of the current storage environment of the video, first obtain the target snapshot type of the backup data and the target writing method of the source data. In this embodiment, there is no specific limitation on what specific snapshot method the storage environment adapts to for data backup, and what method is used to write the source data in the source volume. That is to say, the target snapshot type can be any one of the common snapshot types. For example, the target snapshot type is copy-on-write (COW), redirect-on-write (ROW) or continuous data protection (CDP), and the target writing method of the source data can also be any one of the common data writing methods used in the source volume. For example, the target data writing method can be random write, sequential write or file system write.

[0037] In a specific implementation manner of this application, determining the target snapshot type of the backup data and the target writing method of the source data includes:

[0038] Use the query interface or configuration file of the storage system to determine the target snapshot type;

[0039] Use the command line tool or application log to determine the target writing method.

[0040] Specifically, in practical applications, the storage environment can be detected, and information such as storage configuration and I / O mode can be dynamically analyzed through scripts to determine the target snapshot type and the target writing method.

[0041] For example: SNAPSHOT_TYPE=$(get_snapshot_type) / / Detect the snapshot type through the storage system API or configuration file. Assume the return values are ROW / COW / CDP, that is, the target snapshot type can be ROW, COW, or CDP; WRITE_MODE=$(detect_io_pattern) / / Detect the current writing mode through iostat or application logs. Assume the return values are sequential (continuous) / random / filesystem, that is, the target writing method can be continuous writing, sequential writing, or file system writing.

[0042] S102. Obtain decision-making information recording the snapshot type, writing method, and verification method.

[0043] In this decision-making information, the snapshot type and writing method can be used as conditions, and the verification method is the decision-making result. That is, according to the decision-making logic, judge the verification method to be called, and the relevant conditions and adopted methods of the decision-making logic can be adjusted according to the actual business situation or usage.

[0044] For easy query, this decision-making information can be stored in the form of a chart. As shown in the following table:

[0045] S103. Use the decision-making information to determine the target verification method that matches both the target snapshot type and the target writing method.

[0046] After determining the target snapshot type and the target writing method, the target verification method can be determined using the decision-making information. That is, based on the decision-making information, find the target verification method that adapts to the target snapshot type and the target writing method.

[0047] It should be noted that in this embodiment, when multiple different snapshots are mixed in the storage environment for data protection, the determined target verification method can be one or more. When performing data verification, the corresponding verification process is executed according to the verification method corresponding to the backup data. For example, the data in source volume 1 is verified using the dd command, and the data in source volume 2 is verified using the vdbench data verification mode.

[0048] In a specific implementation manner of this application, using the decision-making information to determine the target verification method that matches both the target snapshot type and the target writing method includes:

[0049] Determine the target backup style using the target snapshot type and the target write method;

[0050] Find the target verification method that matches the target backup style from the decision-making information; among them, the decision-making information includes multiple different backup styles and the mapping relationship between the backup style and the verification method.

[0051] For ease of description, the above steps will be combined and described below.

[0052] In this application, the mapping relationship between different backup styles and verification methods can be directly established in the decision-making information. The backup style is determined based on the snapshot type and the write method. Specifically, a specific snapshot type and a write method can correspond to a backup style. Different backup styles are preset with the best-matching verification methods. In this way, after obtaining the target snapshot type and the target write method, the target backup style corresponding to the current storage environment can be determined. Then, directly query from the decision-making information the verification method that has a mapping relationship with this backup style, and the verification method obtained by this query is the target verification method.

[0053] S104. Execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

[0054] In this embodiment, different verification processes can be preset for different verification methods. After clarifying the target verification method, execute the verification process corresponding to this target verification method to perform consistency verification on the backup data and the source data.

[0055] For the verification that finds the backup data and the source data are consistent, no processing operation is required. When it is found that the backup data and the source data are inconsistent, it can be recorded in the log to facilitate finding and locating the error data and performing error reporting processing.

[0056] Illustrate with an example: If the snapshot type is ROW and the data write is sequential write, it is recommended to call the dd command for hash verification comparison; execute as follows:

[0057] if [ "$METHOD" == "dd_full_hash" ]; then dd if= / dev / source bs=4M |sha256sum > source.hash dd if= / dev / snapshot bs=4M | sha256sum > snap.hashdiff source.hash snap.hash || alert "Block data is inconsistent!"

[0058] In a specific implementation manner of the present application, execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data, including:

[0059] If the target verification method is block device hash verification, obtain the source device information storing the source data and the snapshot device information storing the backup data;

[0060] Utilize the source device information and the snapshot device information to read the source data and the backup data in blocks;

[0061] Calculate the source block hash value of the read source data blocks and the backup block hash value of the read backup data respectively;

[0062] Compare whether the source block hash value is consistent with the backup block hash value;

[0063] If so, determine that the source data and the backup data are consistent;

[0064] If not, determine that the source data and the backup data are inconsistent.

[0065] For the convenience of description, the above steps will be combined and described below.

[0066] Use the dd command (a command for copying and converting data) to perform addressing hash comparison on the block device. By directly reading the specific offset data blocks of the original block device (such as a disk, an LVM logical volume, or a snapshot volume) and calculating their hash values, data consistency is verified.

[0067] Please refer to Figure 2 , for the specific implementation process, including:

[0068] 1. Ensure that the data has been flushed to disk and writing has stopped before creating the snapshot;

[0069] 2. Obtain the device information and parse the device information and size of the source device and the backup device;

[0070] For example, the source device is / dev / vg / original and the backup device is / dev / vg / snap;

[0071] lsblk -b / dev / vg / original # View the total size of the block device;

[0072] blockdev --getbsz / dev / vg / original # Obtain the physical block size of the block device.

[0073] 3. Customize the verification data block size, such as 4K, 8K, 16K, 1M, etc.;

[0074] 4. Customize the address offset of the verification data according to the write address offset. For example, if the write seek = 1, define the verification offset skip = 1.

[0075] 5. Read the source block data and the backup block data from the source device and the backup device respectively, and calculate the hash values.

[0076] 6. Compare whether the source block hash value is the same as the backup block hash value.

[0077] 7. If the comparison results are the same, the verification passes; if the results are different, the verification fails.

[0078] In a specific implementation manner of the present application, comparing whether the source block hash value is the same as the backup block hash value includes: comparing whether the source block hash value is the same as the backup block hash value block by block in a loop. To avoid memory overflow caused by reading a large block device at one time, verification can be performed block by block in fixed segments.

[0079] For example: # Compare block by block in a loop;

[0080] for ((i = 0; i < $TOTAL_BLOCKS; i++)); do;

[0081] dd if = $BLOCK_DEVICE bs = $BLOCK_SIZE count = 1 skip = $i; 2> / dev / null | sha256sum > original_hash.tmp # Read the i-th block data of the original device and calculate the hash;

[0082] Among them, BLOCK_DEVICE is the source volume device number; SNAPSHOT_DEVICE is the snapshot device number; bs is the set block size, count is the number of reads. For example, read 1 time for a 4k block; skip is to start copying and reading after skipping i blocks from the beginning of the source volume device; original_hash.tmp is the file to save the hash value calculated after reading the block data of the source volume; snap_hash.tmp is the file to save the hash value calculated after reading the block data of the snapshot volume.

[0083] dd if = $SNAPSHOT_DEVICE bs = $BLOCK_SIZE count = 1 skip = $i; 2> / dev / null | sha256sum > snap_hash.tmp # Read the i-th block data of the snapshot device and calculate the hash;

[0084] if! cmp -s original_hash.tmp snap_hash.tmp; then # Compare the hashes;

[0085] echo "Block $i is inconsistent!";

[0086] exit 1;

[0087] fi.

[0088] In addition, if the test device resources are running, concurrent processing can be started to accelerate the comparison in parallel. Specifically, it can be processed in slices. Taking 10 threads as an example, it is described as follows:

[0089] seq 0 $TOTAL_BLOCKS | parallel -j10 'dd if=$BLOCK_DEVICE ; bs=$BLOCK_SIZE count=1 skip={} 2> / dev / null | sha256sum > original_{}.tmp'

[0090] seq 0 $TOTAL_BLOCKS | parallel -j10 'dd if=$SNAPSHOT_DEVICE bs=$BLOCK_SIZE count=1 skip={} 2> / dev / null | sha256sum > snap_{}.tmp';

[0091] That is, the command is run in 10 threads.

[0092] Batch comparison can also be performed, and the implementation process is as follows:

[0093] for i in {0..$TOTAL_BLOCKS}; do diff original_${i}.tmp snap_${i}.tmp|| echo "Block $i is inconsistent!"; done.

[0094] In a specific implementation manner of this application, the verification process corresponding to the target verification method is executed to perform consistency verification on the backup data and the source data, including:

[0095] If the target verification method is hierarchical verification, logical integrity verification is performed on the metadata of the backup data;

[0096] If the metadata passes the verification, it is determined that the source data and the backup data are consistent;

[0097] If the metadata fails the verification, the backup data is mounted, and the backup file hash corresponding to the backup data is generated;

[0098] Obtain the source file hash of the source data, and compare whether the backup file hash is the same as the source file hash;

[0099] If so, determine that the source data and the backup data are consistent;

[0100] If not, determine that the source data and the backup data are inconsistent.

[0101] For ease of description, the above steps will be combined and described below.

[0102] Considering the logical integrity of the metadata, the corresponding data content is usually complete. Therefore, in this embodiment, in order to accelerate the verification, the integrity of the metadata logic is verified first. In addition, the metadata (inode, superblock, log, etc.) formats of the same file system (such as ext4 and XFS) are different, and it is meaningless to directly compare the block hashes. Therefore, in this embodiment, the verification is directly focused on the file system layer and the file content layer, and the file content is directly verified. Thus, the accuracy of the consistency verification result is guaranteed.

[0103] Please refer to Figure 3 , and use the hierarchical verification of the file system: when the source volume uses different file systems (such as ext4, XFS, NTFS, etc.), the data consistency verification of the CDP snapshot needs to consider the impact of file system differences on data layout and metadata; the metadata (inode, superblock, log, etc.) formats of different file systems (such as ext4 and XFS) are different, and it is meaningless to directly compare the block device hashes. Therefore, hierarchical verification is adopted, skipping the comparison of the block device layer, and focusing on the verification of the file system layer and the file content layer.

[0104] For the specific implementation, first, perform the file system integrity verification. For each file system type, use the native tool to check the snapshot metadata consistency. The following is an example:

[0105] For the ext4 file system, fsck.ext4 -n / dev / snap_ext4 # -n means read-only check;

[0106] fsck.ext4 is a command used to check and repair the Linux ext2 / ext3 / ext4 file system;

[0107] -n means only check without repair, and is used to preview possible errors;

[0108] / dev / snap_ext4 specifies the snapshot volume to be checked;

[0109] For the xfs file system, xfs_repair -n / dev / snap_xfs # -n means do not modify the file system;

[0110] xfs_repair -n checks whether the file system is damaged; / dev / snap_xfs refers to the snapshot volume to be checked;

[0111] ntfsfix -n / dev / snap_ntfs;

[0112] Among them, ntfsfix –n checks the specified NTFS partition, analyzes whether there are errors in this partition; does not repair the errors, only outputs the inspection results; / dev / snap_ntfs refers to the snapshot volume partition to be checked.

[0113] If the snapshot file system passes the check (no serious errors), it means that the metadata logic is complete. At this time, it can be determined that the snapshot passes the consistency check. Otherwise, the file content is verified.

[0114] Among them, for file content verification, regardless of the type of file system, the hash of the file content can be used as a verification method for file system consistency verification. The specific implementation is to first mount the snapshot, and then generate the file hash. The following is an example:

[0115] mkdir -p / mnt / snap_ext4;

[0116] mount -t ext4 / dev / snap_ext4 / mnt / snap_ext4;

[0117] find / mnt / snap_ext4 -type f -exec sha256sum {} \; > snap_ext4_hashes.txt # Traverse all files to generate hashes.

[0118] Then, compare with the original data hash; the hash comparison needs to be based on the same file path and content, regardless of the file system. If all file hashes are the same, the user data content is the same. The following is an example:

[0119] diff original_hashes.txt snap_ext4_hashes.txt.

[0120] In a specific implementation manner of the present application, the verification process corresponding to the target verification method is executed to perform consistency verification on the backup data and the source data, including:

[0121] If the target verification method is data simulation verification, then use the target tool for storing tests and data verification to read the backup data and generate the data verification value of the backup data;

[0122] Extract the data verification value of the source data recorded during the process of writing the source data using the target tool from the log information;

[0123] Compare whether the data verification value of the backup data is consistent with the data verification value of the source data;

[0124] If so, determine that the source data and the backup data are consistent;

[0125] If not, determine that the source data and the backup data are inconsistent.

[0126] For ease of description, the above steps will be combined and described below.

[0127] Please refer to Figure 4 , taking the target tool vdbench as an example, this embodiment will be described. Vdbench is a tool for storage performance testing and data verification, supporting multiple I / O modes, including random and sequential read and write; when writing a large amount of random block data in real time to the source volume, using the vdbench tool for verification can verify the data consistency of the snapshot.

[0128] The vdbench data verification workflow is as follows: The first write operation in each storage system is recorded in a table. Assuming the block size of the write operation is 1m, two 8-byte logical byte addresses (LBAs) and a byte of data verification key value (i.e., the data verification value, which can mark the number of the write, ranging from 0 - 125, 00 represents creating a write, 01 represents the first overwrite write, and so on. When reaching 126, it turns back to 00 and records a new round) contained in every 512 bytes of this block size will be recorded, and this process will generate a verification log;

[0129] The parameters for opening vdbench to verify data are -v or -j; when using the -v parameter, the generated verification log is directly saved in memory, and -v is directly recorded in memory, which is faster. However, if the storage system restarts or the memory is cleared, the verification log recorded by the -v parameter will be lost;

[0130] When using the -j parameter, a verification log file will be generated. When verifying for the second time, -jr can be used to perform log recovery for verification; -j writes directly to the disk, which is safe but slower. You can choose -jn to write to the disk asynchronously, which has certain guarantees for both speed and safety;

[0131] It is possible to configure vdbench to generate a random write load and enable data verification; create a vdbench configuration file (such as cdp_test.conf), enable the data verification function, and ensure that a verifiable random pattern is generated when writing data. The following is an example:

[0132] fsd=fsd1,anchor= / mnt / source_volume,size=100G,format=yes # Define file system parameters;

[0133] fwd=fwd1,fsd=fsd1,operation=write,xfersize=4k,fileio=random,threads=16 # Define the write workload (random block writes);

[0134] rd=rd1,fwd=fwd1,fwdrate=max,format=restart,data_validation=yes,elapsed=24h,interval=10# Enable data validation (generate verifiable random data);

[0135] Among them, data_validation=yes is the data verification switch, which generates a unique data pattern during writing; other xfersize, fileio, etc. define the data block size, and the writing method is random or sequential, etc.

[0136] Then, call to start vdbench to continuously write. vdbench will continuously write random data to the source volume ( / mnt / source_volume) and record the data pattern. For example: vdbench -f cdp_test.conf -jn -o vdbench_output# Record the write log.

[0137] Snapshots can be triggered at regular intervals, such as configuring a timed snapshot to be taken once every minute.

[0138] The timed snapshot can be mounted to the test directory, configure the vdbench verification task, and create a verification file such as validate.conf; the example is as follows:

[0139] mkdir -p / mnt / snapshots / snap_$TIMESTAMP;

[0140] mount / dev / vg / snap_$TIMESTAMP / mnt / snapshots / snap_$TIMESTAMP # Mount the snapshot;

[0141] fsd=fsd2,anchor= / mnt / snapshots / snap_$TIMESTAMP,depth=1;

[0142] fwd=fwd2,fsd=fsd2,operation=read,xfersize=4k,fileio=random,threads=8 # Define the read-only verification task;

[0143] rd=rd2,fwd=fwd2,fwdrate=max,data_validation=yes,elapsed=1h # Enable data validation (comparison mode);

[0144] When running the verification, vdbench will read the snapshot data and verify whether its pattern matches the expectation during writing. If the data is inconsistent, an error will be reported in the log. For example: vdbench -f validate.conf -jro -cn -o validation_output_$TIMESTAMP.

[0145] In actual implementation, the above steps can also be integrated and automated. Through automated snapshot creation, mounting, and verification, examples are as follows:

[0146] while read -r snap_name timestamp; do;

[0147] mount / dev / vg / $snap_name / mnt / snapshots / $snap_name # Mount the snapshot;

[0148] echo "fsd=fsd2,anchor= / mnt / snapshots / $snap_name,width=1,depth=1" >validate.conf # Generate the verification configuration file; / / / fsd specifies the file system definition name; anchor refers to the directory in which the directory structure will be created; width refers to the number of directories created under the specified directory; depth refers to the number of file levels created under the directory; echo"fwd=fwd2,fsd=fsd2,operation=read,xfersize=4k,fileio=random,threads=8" >> validate.conf; / / / fwd represents the name of the file system workload definition; operation defines the operation performed on the file, which is read-only here; fileio represents the execution method of file io, which is random here; xfersize represents the data size processed by the read and write operations; threads represents the number of concurrent threads of the workload.

[0149] echo "rd=rd2,fwd=fwd2,fwdrate=max,format =yes,elapsed=1h" >>validate.conf; / / / rd represents the running name; fwdrate refers to the number of file system operations executed per second, which is set to the maximum value here; format represents the operations to be performed before starting the run; elapsed represents the running duration.

[0150] vdbench -f validate.conf -jro validation_$timestamp.log # Run the validation;

[0151] if grep -q "Data validation failed" validation_$timestamp.log; then # Check the validation result;

[0152] echo "ERROR: Snapshot $snap_name validation failed!" # If "Data validation failed" is found, the validation fails;

[0153] else echo "Snapshot $snap_name validated successfully." # If "Data validation failed" is not found, the validation is successful;

[0154] fi umount / mnt / snapshots / $snap_name # Unmount and delete the snapshot;

[0155] lvremove -f / dev / vg / $snap_name done < <(tail -f snapshot.log) # Continuously monitor the snapshot log.

[0156] In a specific implementation manner of this application, it further includes:

[0157] Record the validation logs for the validation process and results in a unified log format;

[0158] Provide an external interface for querying the validation logs.

[0159] That is to say, regardless of the type of snapshot and the write method of the source data, and which verification method is selected, when recording the verification log, a unified log format can be adopted to record the verification process and results, and provide an external query interface for the verification log. In this way, when analyzing faults or data protection, a set of logic can be used to analyze the verification log, thereby reducing the related R & D investment and improving the adaptability to different scenarios.

[0160] Applying the snapshot verification method provided by the embodiments of the present application, when backing up the source data based on the snapshot, first determine the target snapshot type of the backup data and the target write method of the source data. For different snapshot types and write methods of metadata, to achieve the consistency verification between the source data and the backup data, the corresponding verification method needs to be changed. In this application, in order to flexibly handle the snapshot consistency verification in different storage scenarios, the verification methods corresponding to different snapshot types and different write methods can be preset for decision-making settings and recorded in the decision information. After obtaining the target snapshot type and the target write method, the decision information can be obtained, and thus the currently adapted target verification method can be determined. Then, execute the verification process corresponding to the target verification method, and the consistency verification between the backup data and the source data can be performed.

[0161] The technical effects of this application: Through automated integration, it is possible to automatically select verification strategies according to different write modes, snapshot types, etc., which can solve the problem of insufficient environmental adaptability during data verification; combining multiple verification methods can cover tests from multiple dimensions and eliminate the blind spots of data verification; when data inconsistency occurs, the root cause can be quickly located.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0163] Please refer to Figure 5 , the embodiments of the present application also provide a snapshot verification device, which includes:

[0164] A snapshot parameter determination module 101, configured to determine the target snapshot type of the backup data and the target write method of the source data when backing up the source data based on the snapshot;

[0165] A decision information acquisition module 102, configured to acquire decision information recording snapshot types, write methods, and verification methods;

[0166] A verification method determination module 103, configured to use the decision information to determine a target verification method that matches both the target snapshot type and the target write method;

[0167] The verification execution module 104 is configured to execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

[0168] When applying the device provided by the embodiments of the present application to back up the source data based on a snapshot, first determine the target snapshot type of the backup data and the target writing method of the source data. For different snapshot types and the writing method of metadata, to achieve the consistency verification between the source data and the backup data, the corresponding verification method needs to be changed. In the present application, in order to flexibly handle the snapshot consistency verification in different storage scenarios, the verification methods corresponding to different snapshot types and different writing methods can be preset for decision-making settings and recorded in the decision information. After obtaining the target snapshot type and the target writing method, the decision information can be obtained, so as to determine the currently adapted target verification method. Then, execute the verification process corresponding to the target verification method, and the consistency verification of the backup data and the source data can be performed.

[0169] Technical effects of the present application: Through automated integration, it is possible to automatically select verification policies according to different writing modes, snapshot types, etc., which can solve the problem of insufficient environmental adaptability during data verification; combining multiple verification methods can cover tests from multiple dimensions and eliminate the blind spots of data verification, so as to quickly locate the root cause when data inconsistency occurs.

[0170] In a specific embodiment of the present application, the verification execution module is specifically configured to, if the target verification method is block device hash verification, obtain the source device information storing the source data and the snapshot device information storing the backup data;

[0171] Read the source data and the backup data in blocks by using the source device information and the snapshot device information;

[0172] Calculate the source block hash value of the read source data block and the backup block hash value of the read backup data respectively;

[0173] Compare whether the source block hash value is consistent with the backup block hash value;

[0174] If so, determine that the source data and the backup data are consistent;

[0175] If not, determine that the source data and the backup data are inconsistent.

[0176] In a specific embodiment of the present application, the verification execution module is specifically configured to compare whether the source block hash value is consistent with the backup block hash value block by block in a loop.

[0177] In a specific embodiment of the present application, the verification execution module is specifically configured to perform a logical integrity verification on the metadata of the backup data if the target verification method is hierarchical verification;

[0178] If the metadata passes the verification, it is determined that the source data and the backup data are consistent;

[0179] If the metadata fails the verification, mount the backup data and generate a backup file hash corresponding to the backup data;

[0180] Obtain the source file hash of the source data and compare whether the backup file hash is consistent with the source file hash;

[0181] If so, it is determined that the source data and the backup data are consistent;

[0182] If not, it is determined that the source data and the backup data are inconsistent.

[0183] In a specific embodiment of the present application, the verification execution module is specifically configured to, if the target verification method is data simulation verification, use the target tool for storing tests and data verification to read the backup data and generate a data verification value of the backup data;

[0184] Extract from the log information the data verification value of the source data recorded during the process of writing the source data using the target tool;

[0185] Compare whether the data verification value of the backup data is consistent with the data verification value of the source data;

[0186] If so, it is determined that the source data and the backup data are consistent;

[0187] If not, it is determined that the source data and the backup data are inconsistent.

[0188] In a specific embodiment of the present application, the snapshot parameter determination module is specifically configured to use the query interface or configuration file of the storage system to determine the target snapshot type;

[0189] Use the command line tool or application log to determine the target writing method.

[0190] In a specific embodiment of the present application, the verification method determination module is specifically configured to use the target snapshot type and the target writing method to determine the target backup style; find the target verification method that matches the target backup style from the decision-making information; wherein, the decision-making information includes various different backup styles and the mapping relationship between the backup styles and the verification methods.

[0191] For the description of the features in the embodiment corresponding to the snapshot verification device, reference can be made to the relevant description of the embodiment corresponding to the snapshot verification method, which will not be elaborated here one by one.

[0192] Corresponding to the above method embodiments, an embodiment of the present application further provides an electronic device. An electronic device described below can be correspondingly referred to the snapshot verification method described above.

[0193] See Figure 6 As shown, the electronic device includes:

[0194] A memory 332 for storing computer programs;

[0195] A processor 322 for implementing the steps of the snapshot verification method in the above method embodiment when executing the computer program.

[0196] Specifically, please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device may vary greatly due to configuration or performance, and may include one or more processors (central processing units, CPUs) (for example, one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be transient storage or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the data processing device. Further, the processor 322 can be set to communicate with the memory 332 and execute a series of instruction operations in the memory 332 on the electronic device 301.

[0197] The electronic device 301 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0198] The steps in the snapshot verification method described above can be implemented by the structure of the electronic device.

[0199] Corresponding to the above method embodiments, an embodiment of the present application further provides a readable storage medium. A readable storage medium described below can be correspondingly referred to the snapshot verification method described above.

[0200] An embodiment of the present application further provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and the computer program is set to execute the steps in any of the above snapshot verification method embodiments when running.

[0201] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0202] The embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the snapshot verification method.

[0203] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the snapshot verification method.

[0204] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0205] The above has introduced in detail a snapshot verification method, device, electronic device, and readable storage medium provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A snapshot verification method, characterized in that, Including: When backing up source data based on a snapshot, determining the target snapshot type of the backup data and the target writing method of the source data; Obtaining decision information recording snapshot type, writing method, and verification method; Using the decision information to determine a target verification method that matches both the target snapshot type and the target writing method; Executing the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

2. The snapshot verification method according to claim 1, wherein Executing the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data, including: If the target verification method is block device hash verification, obtaining source device information storing the source data and snapshot device information storing the backup data; Using the source device information and the snapshot device information to read the source data and the backup data in blocks; Calculating the source block hash value of the read source data blocks and the backup block hash value of the read backup data respectively; Comparing whether the source block hash value is consistent with the backup block hash value; If so, determining that the source data and the backup data are consistent; If not, determining that the source data and the backup data are inconsistent.

3. The snapshot verification method according to claim 2, wherein Comparing whether the source block hash value is consistent with the backup block hash value, including: Comparing whether the source block hash value is consistent with the backup block hash value block by block in a loop.

4. The snapshot verification method according to claim 1, characterized in that, Executing the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data, including: If the target verification method is hierarchical verification, performing logical integrity verification on the metadata of the backup data; If the metadata passes the verification, determining that the source data and the backup data are consistent; If the metadata fails the verification, mounting the backup data and generating a backup file hash corresponding to the backup data; Obtaining the source file hash of the source data and comparing whether the backup file hash is consistent with the source file hash; If so, determining that the source data and the backup data are consistent; If not, determining that the source data and the backup data are inconsistent.

5. The snapshot verification method according to claim 1, wherein Executing the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data, including: If the target verification method is data simulation verification, using a target tool for storing tests and data verification to read the backup data and generate a data verification value of the backup data; Extracting from the log information the data verification value of the source data recorded during the process of writing the source data using the target tool; Comparing whether the data verification value of the backup data is consistent with the data verification value of the source data; If so, determining that the source data and the backup data are consistent; If not, determining that the source data and the backup data are inconsistent.

6. The snapshot verification method according to claim 1, wherein Determining the target snapshot type of the backup data and the target writing method of the source data, including: Using the query interface or configuration file of the storage system to determine the target snapshot type; Using the command line tool or application log to determine the target writing method.

7. The snapshot verification method according to any one of claims 1 to 6, characterized in that Determine a target verification method that matches both the target snapshot type and the target writing method by using the decision information, including: Determine a target backup style by using the target snapshot type and the target writing method; Find a target verification method that matches the target backup style from the decision information; wherein, the decision information includes multiple different backup styles and the mapping relationship between the backup styles and the verification methods.

8. A snapshot verification device, characterized in that Including: A snapshot parameter determination module, configured to determine the target snapshot type of the backup data and the target writing method of the source data when backing up the source data based on a snapshot; A decision information acquisition module, configured to acquire decision information recording snapshot types, writing methods, and verification methods; A verification method determination module, configured to determine a target verification method that matches both the target snapshot type and the target writing method by using the decision information; A verification execution module, configured to execute the verification process corresponding to the target verification method to perform consistency verification on the backup data and the source data.

9. An electronic device, characterized in that, Including: A memory, configured to store a computer program; A processor, configured to implement the steps of the snapshot verification method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the snapshot verification method according to any one of claims 1 to 7.

Images