File data verification method and device, computer equipment and readable storage medium

Abstract

The invention relates to a file data verification method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list; performing file operation of at least one operation type on the test file to obtain an operated file, performing name list updating operation on the test file name list, and performing Hash mapping table updating operation on the Hash mapping table; for any operated file, calculating actual verification data of any data block, and querying a mapping value of any data block from the updated Hash mapping table to obtain expected verification data of any data block; and under the condition that the actual verification data is consistent with the expected verification data, determining that the data consistency verification result of the file after any operation is passed. By adopting the method, the consistency verification accuracy of the file data can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer storage technology, and in particular to a file data verification method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] The consistency check of file data can be to check the integrity and consistency of the data stored in the file during the processing process. For example, after performing file operations such as creation, writing, and truncation on the data stored in the file, it can be checked whether the data content is consistent with expectations. The current file data verification method can only support data consistency verification after reading and writing operations on file data, and cannot support a variety of data consistency verification scenarios, resulting in a low accuracy rate of file data consistency verification in the face of various file operations other than reading and writing operations. Summary of the invention

[0003] Based on this, it is necessary to provide a file data verification method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of file data consistency verification in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for verifying file data, comprising:

[0005] Obtain a list of test file names corresponding to each test file and a hash mapping table corresponding to the list of test file names; the hash mapping table includes key values ​​of data blocks in each of the test files and corresponding mapping values; the key values ​​are determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block;

[0006] Performing at least one type of file operation on the test file to obtain an operated file, performing a name list update operation matching the operation type on the test file name list to obtain an updated name list, and performing a hash map update operation matching the operation type on the hash map to obtain an updated hash map;

[0007] For any of the post-operation files in the updated name list, calculate actual verification data of any data block in any of the post-operation files, and query a mapping value corresponding to a key value of any of the data blocks from the updated hash mapping table to obtain expected verification data of any of the data blocks;

[0008] When the actual verification data is consistent with the expected verification data, it is determined that the data consistency verification result of any of the operated files is verification passed.

[0009] In one embodiment, the updated name list further includes the number and size of data blocks included in each of the post-operation files; before calculating the actual check data of any data block in any of the post-operation files, the method further includes:

[0010] In the case of successfully reading any of the post-operation files, calculating the actual file size of any of the post-operation files, and obtaining the expected file size of any of the post-operation files; the expected file size is calculated based on the number of data blocks and the data block size included in any of the post-operation files;

[0011] When the actual file size is consistent with the expected file size, return to the step of calculating the actual check data of any data block in any of the operated files.

[0012] In one embodiment, the data content of the data block in the test file is generated by a random seed; the mapping value also includes the value of the random seed; after obtaining the expected verification data of any of the data blocks, the method further includes:

[0013] In the case where the actual verification data does not conform to the expected verification data, querying the value of the random seed contained in the corresponding mapping value from the hash mapping table according to the key value of any of the data blocks;

[0014] According to the value of the random seed, the data content of any of the data blocks is restored, and a data consistency check result of any of the files after the operation is determined to be a check failure.

[0015] In one of the embodiments, when the file operation is a link operation, the hash map update operation is not to update the hash map; the post-operation file is a link file created based on the test file, and the link file has the same data content and index identifier as each data block of the test file in a one-to-one correspondence; the name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, including:

[0016] Add the file name of the link file to the test file name list, and save the number of data blocks and the size of the data blocks included in the link file to the test file name list to obtain the updated name list;

[0017] The file name of the link file is composed of a file prefix name and a file suffix name of the link file, and the file prefix name of the link file is consistent with the file prefix name of the test file.

[0018] In one of the embodiments, when the file operation is a rename operation, the hash map update operation is not to update the hash map; the post-operation file is a renamed file with a new file name obtained by modifying the file prefix name of the test file, and the renamed file has the same data content and index identifier as each data block of the test file in a one-to-one correspondence; the name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, including:

[0019] Modify the file prefix names of the test files in the test file name list to the new file names to obtain the updated name list;

[0020] The new file name is composed of a file prefix name and a file suffix name of the renamed file, and the file prefix name of the renamed file is consistent with the file prefix name of the test file.

[0021] In one of the embodiments, when the file operation is a truncation operation, the post-operation file includes a truncated file obtained by truncating a number of data blocks of the test file; and performing a hash mapping table update operation matching the operation type on the hash mapping table to obtain an updated hash mapping table includes:

[0022] Generate a key value to be deleted according to the file prefix name of the test file and the index identifier of the data block truncated in the test file;

[0023] The key value to be deleted and the mapping value corresponding thereto are deleted from the hash mapping table to obtain the updated hash mapping table.

[0024] In one of the embodiments, the data content of the data block in the test file is generated by a random seed; when the file operation is an extension operation, the post-operation file includes an extended file obtained after inserting a plurality of data blocks into the test file; and performing a hash mapping table update operation matching the operation type on the hash mapping table to obtain an updated hash mapping table includes:

[0025] Calculating the check data of the data block inserted in the test file as the expected check data of the inserted data block, and obtaining the value of the random seed corresponding to the inserted data block;

[0026] Determining a mapping value of the inserted data block according to expected check data corresponding to the inserted data block and a value of a random seed;

[0027] Generate a key value of the inserted data block according to the file prefix name of the test file and the index identifier of the inserted data block in the extended file;

[0028] Inserting the mapping value of the inserted data block and the key value of the inserted data block into the hash mapping table to obtain the updated hash mapping table.

[0029] In a second aspect, the present application also provides a file data verification device, comprising:

[0030] An acquisition module, used to acquire a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list; the hash mapping table includes key values ​​of data blocks in each test file and corresponding mapping values; the key value is determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block;

[0031] an operation update module, configured to obtain an operated file by performing at least one operation type of file operation on the test file, and to obtain an updated name list by performing a name list update operation matching the operation type on the test file name list, and to obtain an updated hash map by performing a hash map update operation matching the operation type on the hash map;

[0032] A data acquisition module, for calculating actual verification data of any data block in any of the post-operation files in the updated name list, and querying a mapping value corresponding to a key value of any of the data blocks from the updated hash mapping table to obtain expected verification data of any of the data blocks;

[0033] The verification module is used to determine that the data consistency verification result of any of the files after the operation is a verification pass when the actual verification data is consistent with the expected verification data.

[0034] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0037] The above-mentioned file data verification method, device, computer equipment, computer-readable storage medium and computer program product obtain a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list, wherein the hash mapping table includes a key value of a data block in each test file and a corresponding mapping value, the key value is determined according to a file prefix name of the test file to which the data block belongs and an index identifier of the data block in the test file to which it belongs, and the mapping value includes expected verification data of the data block; by performing at least one type of file operation on the test file, a post-operation file is obtained, and a name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, and a hash mapping table update operation matching the operation type is performed on the hash mapping table to obtain an updated hash mapping table; for any post-operation file in the updated name list, the actual verification data of any data block in any post-operation file is calculated, and the mapping value corresponding to the key value of any data block is queried from the updated hash mapping table to obtain the expected verification data of any data block; when the actual verification data is consistent with the expected verification data, the data consistency verification result of any post-operation file is determined to be verification passed. The file prefix name and data block index identifier of the test file are used to generate a unique key value, so that the file data can accurately locate the data block and maintain the corresponding relationship of the verification data after performing various file operations; by updating the test file name list and the hash mapping table, the changes in the file data after the file operation are dynamically adapted, and the comparison of the actual verification data with the expected verification data is combined to ensure the accuracy and real-time performance of the consistency verification process, overcome the limitation of the traditional method that only consistency verification can be performed on read and write operations, and significantly improve the verification accuracy and reliability in complex file operation scenarios. By introducing the test file name list and the hash mapping table containing the data block key value and the corresponding expected verification data, accurate verification of data consistency during the file operation process is achieved, which can improve the accuracy of consistency verification of file data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1An application environment diagram of a file data verification method in an embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of a file data verification method in an embodiment;

[0041] Figure 3 A schematic diagram of a test file name list in one embodiment;

[0042] Figure 4 A schematic diagram of a hash mapping table in one embodiment;

[0043] Figure 5 A logic diagram of a file data verification method in an embodiment;

[0044] Figure 6 A schematic diagram of a test file name list supporting link operations in one embodiment;

[0045] Figure 7 A structural block diagram of a file data verification device in an embodiment;

[0046] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The file data verification method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 obtains a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list; the hash mapping table includes key values ​​of data blocks in each test file and corresponding mapping values; the key value is determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block; the terminal 102 obtains a post-operation file by performing at least one type of file operation on the test file, and performs a name list update operation matching the operation type on the test file name list to obtain an updated name list, and performs a hash mapping table update operation matching the operation type on the hash mapping table to obtain an updated hash mapping table; for any post-operation file in the updated name list, calculate the actual verification data of any data block in any post-operation file, and query the mapping value corresponding to the key value of any data block from the updated hash mapping table to obtain the expected verification data of any data block; when the actual verification data is consistent with the expected verification data, determine that the data consistency verification result of any post-operation file is verification passed. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0049] In an exemplary embodiment, Figure 2 As shown, a method for verifying file data is provided, and the method is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation, including:

[0050] Step S202, obtaining a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list.

[0051] In the specific implementation, the test file can be logically divided into several data blocks. When creating each test file, data can be written according to the size and number of data blocks. The writing and verification operations of the test file can use the data block as the basic unit, that is, the file operation of each data block can be regarded as an independent input / output (I / O) operation.

[0052] In one embodiment, writing data to each data block may be filled with a random number generated by a random seed. The random seed may be a number used to initialize a pseudo-random number generator. Given the same initial seed, the pseudo-random number generator will generate the same random number sequence. For example, the random seed may be a number, such as an integer. Different numbers will generate different random number sequences. For example, when the value of the random seed is 42, a random number sequence A may be generated. When the value of the random seed is 43, a random number sequence B may be generated.

[0053] In a specific implementation, the test file may include multiple data blocks, each of which may correspond to a verification data, so that the verification of the file data may be performed in units of data blocks. When writing data to a data block, the verification data may be calculated for the written data according to a certain algorithm. Optionally, the algorithm for calculating the verification data may be MD5 (Message-Digest Algorithm 5), CRC (Cyclic Redundancy Check), etc. Among them, the CRC algorithm is a fast algorithm that generates a short fixed-bit verification code based on the data, which is mainly used to detect or verify errors that may occur after data transmission or storage; the MD5 algorithm is a widely used hash function that can generate a 128-bit hash value to ensure the integrity and consistency of information transmission.

[0054] In the specific implementation, the verification data can be stored separately from the test file, so as to effectively verify the scenario of incomplete data and old data covering new data. The storage method of the verification data can be a hash map (HashMap). The hash map is a data structure based on the hash table. Each element stored in it is a pair of key value (key)-map value (value), that is, a key-value pair. The hash map (HashMap) converts the key value (key) to the corresponding map value (value) through the mapping function Hash (key) to speed up the search.

[0055] The test file name list (File List) may refer to a name list of all test files for this verification, including the name of each test file.

[0056] In a specific implementation, the test file name list can be used to easily verify extreme data loss scenarios such as the loss of the entire file. For example, when reading a test file based on the file name, if the test file cannot be found, it means that the test file is lost. At the same time, it is also convenient to quickly obtain the names of all test files through the test file name list when verifying file data, without traversing the test file directory, thereby improving operational efficiency.

[0057] In one of the embodiments, the test file name list also includes the number of data blocks and the size of data blocks included in each test file, so that the file name of the test file, the data block size of the test file and the number of data blocks of the test file can be stored together in the test file name list. The technical effects of this embodiment include two aspects: first, it is easy to read the test file, because the data block size and the number of data blocks of the test file are known, and the specified number of data blocks can be read from the test file in sequence and their consistency can be verified; second, it can quickly verify whether the size of the test file is correct. For example, the number of data blocks of a test file recorded in the test file name list is 10, but the actual number of data blocks obtained for the test file is 8, which means that the test file has lost 2 data blocks of data. At this time, there is no need to perform the subsequent steps of reading the data content of the test file, and the data consistency verification result can be directly determined as verification failure, thereby improving the efficiency of data consistency verification.

[0058] To facilitate the understanding of those skilled in the art, Figure 3 A schematic diagram of a test file name list is exemplarily provided.

[0059] In summary, separately saving the file name of the test file and its data block size and number of data blocks through the test file name list (linked list or array) has the following beneficial effects: it can easily verify the scenario of the entire file being lost, the list saves all file names, and when a file does not exist when searching according to the file name, it indicates that a file is lost; based on the saved data block size and number of data blocks, it is possible to quickly verify the scenario of inconsistent file sizes, for example, the expected file size after a file is truncates is smaller, but if a program fault bug causes the read file to still be the original initial size, then the data inconsistency problem can be quickly verified; it is convenient to quickly obtain all file names when verifying data, without traversing the file directory, thereby improving operation efficiency; it is convenient to expand and save more types of file metadata in the future to support metadata consistency verification.

[0060] Among them, the hash mapping table includes the key value of the data block in each test file and the corresponding mapping value; the key value is determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes the expected verification data of the data block.

[0061] Among them, the file prefix name (base name) of the test file can refer to the name prefix of the test file. When a test file is used as a source file, the file prefix name of the test file can refer to a name prefix common to a link file generated based on the source file, a rename file generated based on the source file, and the source file. Exemplarily, the file name of the test file can be a fixed string plus a monotonically increasing unique identifier (Identity Document, ID). For example, assuming that the file name of the test file is file{ID}, such as file1, file2, the file{ID} here can be the file prefix name (base name), that is, the file name of the initially created test file can only include the file prefix name. When a subsequent link operation, rename operation, or other file operation is performed on a test file to generate a post-operation file, the file name of the post-operation file can be a file suffix name added on the basis of the file prefix name of the test file. Exemplarily, in a scenario where a link operation or a rename operation is performed on a test file, a test file named file{ID} is used as a source file, and a link file and a rename file are created based on the source file. The naming rule of the link file and the rename file can be "file{ID}-{random string}", where file{ID} is the file prefix name, that is, the file name of the source file, and {random string} is equivalent to the file suffix name. In other words, assuming that a link file is created again for a link file, or a rename file is renamed to generate a rename file, or a link file is created for a rename file, the new file will not inherit the file suffix name of its direct source file, but directly use the file prefix name of the source file plus a random string as the file name, that is, "file{ID}-{random string}", to avoid unlimited growth of the file name. For example, the initial source file is file1, which is the file prefix name. The link file created for it is file1-diw923d. The link file created for file1-diw923da is file1-jk9h6ndp. The renamed file generated after renaming file1-jk9h6nd is file1-x8asp3db. They are all in the format of the file prefix name plus a random string.

[0062] It can be seen that this naming method based on file prefix names and the method of storing and managing verification data generated based on file prefix names have the following beneficial effects: all related files generated by link operations and rename operations have the same file content, and the file prefix name naming scheme is adopted, so all related files have the same file prefix name. Since the database index identifiers of all related files generated by link operations and rename operations are the same, and the file prefix names are also the same, the verification data of all related files are also the same. All related files can share a copy of verification data, which simplifies the management of verification data, facilitates the superposition of link operations, rename operations and other operations, and can effectively verify the data consistency of all files. Since the key to the storage of verification data is to effectively support the storage and retrieval of link operations and rename operation scenarios, such as link operations, because the file after the link operation is a new file, but the content is the same as the original file, any modification of any file related to the link operation will affect the remaining files, so the verification data of these link files and source files need to be managed uniformly. Therefore, the file naming method based on the file prefix name and the verification data management method based on the file prefix name can effectively support the data consistency verification scenarios under the link operation and rename operation scenarios, and can friendly support the data consistency verification of the create operation, write operation, append operation, truncate operation, link operation, rename operation, unlink operation, read operation and their superimposed combination scenarios.

[0063] The block index may be an index number of a data block in the test file. For example, if the test file has N data blocks, the range of the block index is from 0 to N-1.

[0064] In the specific implementation, for any data block in the test file, the key value of the data block can be constructed according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block. Since the file prefix name is unique, each test file corresponds to a different file prefix name, and the index identifier is also unique in the test file, the key value (key) is unique.

[0065] Optionally, in order to make the key value (key) hash distribution more uniform, the file prefix name and the index identifier can be combined, and the MD5 algorithm can be used to generate a unique hash value for the combined data as the key value.

[0066] Among them, the expected verification data is the verification data calculated based on the written data when the data is initially written to the data block of the test file. For example, the expected verification data can be a CRC value calculated based on the CRC algorithm. In a specific implementation, a mapping value (value) can be constructed based on the expected verification data. Further optionally, the mapping value of the data block can be constructed based on the value of the random seed and the expected verification data, that is, the mapping value can include the expected verification data of the data block and the value of the random seed for generating the data content of the data block, so that when the subsequent data consistency check result is a verification failure, the data content of the data block can be restored based on the value of the random seed.

[0067] To facilitate the understanding of those skilled in the art, Figure 4 A schematic diagram of a hash mapping table is provided as an example. In a test file with a file prefix name of file1, each data block has a key value and a mapping value stored in the hash mapping table. The key value is an MD5 value calculated by the MD5 algorithm based on the file prefix name (file1) and the data block index identifier (0~N-1), and the mapping value is a value determined based on the expected check data (CRC value) and a random seed (seed).

[0068] In one embodiment, a user may call a create operation to input a test directory, the depth and width of subdirectories in the test directory, the number of test files in the subdirectories, the number of data blocks of each test file, and the size of the data blocks to generate a list of test file names, wherein the sizes of the data blocks in a test file may be the same, and the size of a test file is equal to the number of data blocks included in the test file multiplied by the data block size. The create operation is used to create a test file and write data to the data block, and at the same time, it calculates and stores the expected verification data and the list of test file names, which is the basis for other subsequent file operations. Specifically, the create operation may include the following steps:

[0069] (1) First, create all subdirectories under the test directory, and then create each test file in each subdirectory in turn and write data. Simple rules can be used for file naming, such as a random string of fixed length, or a fixed string plus a monotonically increasing unique identifier (Identity Document, ID). For example, assume that the file name of the test file is file{ID}, such as file1, file2. Here, file{ID} can also be the file prefix name (basename). All newly created test files can use this naming rule, that is, the file name of the initially created test file can only contain the file prefix name. When a subsequent file operation such as linking or renaming is performed on a test file to generate a post-operation file, the file name of the post-operation file can be the file prefix name of the test file with the file suffix name added.

[0070] (2) For each test file, data can be written in sequence according to the number of data blocks and the size of the data blocks included in the test file. Each time a data block is written, verification data is calculated based on the written data as expected verification data, and the expected verification data is stored separately in the hash mapping table. The data of each data block can be filled with a random number generated by a random seed, and the value of the random seed and the expected verification data can be saved together as a mapping value in the hash mapping table, which is used to play back the initially written data according to the random seed in the event that the data consistency check fails in the future.

[0071] (3) After writing each test file, add the file name of the test file to the test file name list. The test file name list can use data structures such as arrays or linked lists. At the same time, store the data block size and number of data blocks of the test file together with the file name of the test file. After the data of all test files are written, the test file name list can be serialized and saved to a file for persistent storage. The file can be saved in the test directory or in the local directory.

[0072] (4) A hash map is a key-value storage format. Each data block corresponds to a key-value pair. The key (key value) is a combination of the file prefix name and the index identifier of the data block. The value (map value) may include the expected verification data of the data block and the value of the random seed. Specifically, a hash map may be created when a create operation is called. Then, each time a data block of a test file is written, the verification data of the data block is calculated as the expected verification data of the data block. Then, the key value and mapping value of the data block are constructed in accordance with the above-mentioned embodiment, and the key value and mapping value are inserted into the hash map. When all test files are written, the hash map saves the verification data of all test files. Then, the hash map is serialized and saved to a file for persistent storage. The file may be stored in a test directory or in a local directory. If the hash map file and the test file name list file are saved together in a local directory, the verification data itself can be ensured not to be affected.

[0073] Step S204, performing at least one operation type of file operation on the test file to obtain a post-operation file, and performing a name list update operation matching the operation type on the test file name list to obtain an updated name list, and performing a hash mapping table update operation matching the operation type on the hash mapping table to obtain an updated hash mapping table.

[0074] To facilitate the understanding of those skilled in the art, Figure 5 A logic diagram of a file data verification method is provided illustratively. Each file operation on a data block in a test file may involve updating of a test file name list and verification data.

[0075] Among them, at least one type of file operation (process) may include a create operation, a write operation, an append operation, a truncate operation, a link operation, a rename operation, an unlink operation, a read operation, and other file operations. Optionally, these file operations may be executed by a user through a command line interface (CLI) by inputting a CLI command, and each file operation may correspond to a CLI command, so that the user can conveniently execute a certain file operation, or combine various file operations through scripts to construct a complex test scenario covering various file operations.

[0076] Each file operation may need to interact with the test file, the test file name list, and the hash map, that is, each file operation may access or modify the test file, the test file name list, and the hash map, so the test file name list and the hash map may need to be updated.

[0077] In a specific implementation, for different operation types, the name list update operation performed on the test file name list is also different, and the name list update operation may include performing an update operation on the test file name list and not performing an update operation on the test file name list. Exemplarily, if the file operation is a rename operation, the name list update operation may include modifying the file name in the test file name list; if the file operation is a link operation, a new link file is created, and the name list update operation may include adding the file name of the new link file to the test file name list; if the file operation is a truncation operation, only a number of data blocks in the test file are truncated, and the file name of the test file is not changed, so the test file name list may not be updated.

[0078] Furthermore, in the case where the test file name list includes the number of data blocks and the size of data blocks included in each test file, the name list update operation also involves an update operation on the number of data blocks and the size of data blocks in the test file name list, that is, assuming that the file operation modifies the number of data blocks or the size of data blocks in the test file, the number of data blocks or the size of data blocks in the test file name list must also be synchronously updated.

[0079] In a specific implementation, for different operation types, different hash mapping table update operations matching the operation types are performed on the hash mapping table. The hash mapping table update operation may include performing an update operation on the hash mapping table and not performing an update operation on the hash mapping table. Exemplarily, if the file operation is a write operation, the data block in the test file will be overwritten with new data content, and new verification data corresponding to the data block can be calculated based on the new data content. Therefore, the hash mapping table update operation may include making corresponding modifications to the expected verification data of the data block recorded in the hash mapping table according to the new verification data to obtain updated expected verification data, that is, updated mapping value, to obtain an updated hash mapping table; if the file operation is a truncation operation, several data blocks in the test file will be truncated, and the hash mapping table update operation may include deleting the key values ​​and mapping values ​​of these truncated data blocks recorded in the hash mapping table to obtain an updated hash mapping table; if the file operation is a rename operation, the renamed file uses the same file prefix name as the source file, and the data block in the test file has not changed, that is, the data block index identifier and the file prefix name have both changed, that is, the key value has not changed, and the expected verification data calculated based on the data content of the data block has not changed, that is, the mapping value has not changed, therefore, the hash mapping table update operation may be not performing an update operation on the hash mapping table.

[0080] Optionally, the hash mapping table update operation may include updating the key value or updating the mapping value, or may include updating both the key value and the mapping value, or may include not updating either the key value or the mapping value.

[0081] Step S206, for any post-operation file in the updated name list, calculate the actual verification data of any data block in any post-operation file, and query the mapping value corresponding to the key value of any data block from the updated hash mapping table to obtain the expected verification data of any data block.

[0082] The terminal may traverse each post-operation file in the updated name list, read each data block of any post-operation file, and calculate actual verification data of the currently read data block.

[0083] The actual verification data is verification data calculated based on data in the test file after performing several file operations on the test file.

[0084] The key value and mapping value recorded in the updated hash mapping table may be the updated value after the hash mapping table update operation is performed. Therefore, the mapping value corresponding to the key value of any data block is queried from the updated hash mapping table to obtain the expected verification data of the data block, and the expected verification data may be the updated value after the hash mapping table update operation is performed.

[0085] Step S208: When the actual verification data is consistent with the expected verification data, it is determined that the data consistency verification result of the file after any operation is verification passed.

[0086] In some common file operation scenarios, randomly combined file operations such as linking, renaming, and unlinking can easily lead to data inconsistency problems. For example, a test file is used as a source file to perform a link operation, multiple link files are created, and then one of the link files is written, truncated, or unlinked, which may affect the file data of the source file or other link files. Therefore, even if the file after the operation is performed on a test file, and the expected verification data in the hash map is updated in time, other file data related to the test file may also change, resulting in inconsistency between the actual verification data and the expected verification data. Therefore, it is necessary to verify whether the actual verification data of each data block in the post-operation file is consistent with the expected verification data.

[0087] If the actual verification data is consistent with the expected verification data, the data consistency verification result of the file after any operation is determined to be verified as passed, that is, the file data is consistent. If the actual verification data is inconsistent with the expected verification data, the data consistency verification result of the file after any operation is determined to be verified as failed, that is, the file data is inconsistent, and the data content in the data block that failed the verification can be exported for subsequent problem analysis and location.

[0088] In some embodiments, multiple types of file operations may be performed on the test file, and these basic file operations may be combined into complex operations to verify data consistency scenarios involving various file operations. Specifically, the verification case may include the following steps:

[0089] S1. The user creates 10,000 1-megabyte test files;

[0090] S2: Select 50% of the test files to perform the link operation;

[0091] S3: Randomly select 50% of the test files to perform append write operations, and the file append ratio is 30%;

[0092] S4: Randomly select 50% of the files from all the test files and perform the rename operation;

[0093] S5: Randomly select 50% of the files from all the test files to perform the expansion operation, and the expansion ratio is 30%;

[0094] S6: Randomly select 50% of the files from all test files to perform truncation, with a truncation ratio of 40%;

[0095] S7: Randomly select 50% of the files from all the test files to perform overwrite operations, with an overwrite ratio of 30%;

[0096] S8: Randomly select 10% of the files from all the test files and perform the unlink operation;

[0097] S9: Read all test files to verify data consistency;

[0098] S10: Repeat steps S2-S9 for 100 times.

[0099] In this test case, since various file operations are covered and each file may have different combinations of operations due to the randomness of file selection, the data consistency of complex scenarios composed of various operation combinations can be verified, thereby solving the data consistency verification problems in file operations such as create, write, append, truncate, link, rename, unlink, and read and their complex combination scenarios. In particular, for scenarios with link and rename operations, the accuracy of data consistency verification in these file operation scenarios can be effectively improved.

[0100] In the above-mentioned file data verification method, a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list are obtained, wherein the hash mapping table includes the key value of the data block in each test file and the corresponding mapping value, the key value is determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs, and the mapping value includes the expected verification data of the data block; by performing at least one operation type of file operation on the test file, a post-operation file is obtained, and a name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, and a hash mapping table update operation matching the operation type is performed on the hash mapping table to obtain an updated hash mapping table; for any post-operation file in the updated name list, the actual verification data of any data block in any post-operation file is calculated, and the mapping value corresponding to the key value of any data block is queried from the updated hash mapping table to obtain the expected verification data of any data block; when the actual verification data is consistent with the expected verification data, it is determined that the data consistency verification result of any post-operation file is a verification pass. The file prefix name and data block index identifier of the test file are used to generate a unique key value, so that the file data can accurately locate the data block and maintain the corresponding relationship of the verification data after performing various file operations; by updating the test file name list and the hash mapping table, the changes in the file data after the file operation are dynamically adapted, and the comparison of the actual verification data with the expected verification data is combined to ensure the accuracy and real-time performance of the consistency verification process, overcome the limitation of the traditional method that only consistency verification can be performed on read and write operations, and significantly improve the verification accuracy and reliability in complex file operation scenarios. By introducing the test file name list and the hash mapping table containing the data block key value and the corresponding expected verification data, accurate verification of data consistency during the file operation process is achieved, which can improve the accuracy of consistency verification of file data.

[0101] In another embodiment, the updated name list further includes the number of data blocks and the size of the data blocks included in each post-operation file; before calculating the actual verification data of any data block in any of the post-operation files, it also includes:

[0102] When any post-operation file is successfully read, the actual file size of any post-operation file is calculated, and the expected file size of any post-operation file is obtained; the expected file size is calculated based on the number of data blocks and the data block size included in any post-operation file; when the actual file size is consistent with the expected file size, return to the step of calculating the actual verification data of any data block in any post-operation file.

[0103] In a specific implementation, the terminal can read the post-operation files through a read operation and verify the data consistency of these post-operation files. Optionally, when the user calls the read operation, the user can enter the number or proportion of files to be verified. The terminal can randomly select the specified number or proportion of post-operation files and perform the read operation, or it can perform the read operation on all post-operation files by default.

[0104] Taking reading all the post-operation files as an example, the terminal can traverse the updated name list, and for each post-operation file, it can first check whether it exists. If it does not exist, it means that data loss has occurred. The way to check whether it exists is to determine whether the post-operation file can be successfully read. For example, if an error message indicating "file path does not exist" appears during reading, it means that the reading failed and the post-operation file does not exist. In the case of successfully reading any post-operation file, the file size (size) is verified. Specifically, the actual file size of the post-operation file is calculated, and then the expected file size of the post-operation file is calculated based on the number of data blocks and data block size included in the post-operation file recorded in the updated name list, and the actual file size is compared with the expected file size. If the actual file size is inconsistent with the expected file size, it means that data inconsistency has occurred. It can be directly judged that the data consistency check result of the post-operation file is failed, and there is no need to verify the data content of the post-operation file. If the actual file size is consistent with the expected file size, the data content of the file after the operation can be verified. At this time, each data block in the file after the operation can be traversed and read. Assuming that the number of data blocks is block number and the data block size is block size, the index identifier range of the data block can be [0, block number-1]. For the data block currently being read, assuming that the index identifier is N, the offset of the file address during reading is N*block size, and the reading length is block size. After the reading is completed, the actual verification data of the data block is calculated, and then the key value of the data block is composed of the file prefix name of the file to be operated to which the data block belongs and the index identifier of the data block. The expected verification data corresponding to the key value is queried from the updated hash mapping table, and the actual verification data is compared with the expected verification data. If the actual verification data is consistent with the expected verification data, the data consistency verification result is verification passed; if the actual verification data is inconsistent with the expected verification data, it means that data inconsistency has occurred, and the data consistency verification result is verification failed.

[0105] The file data verification method proposed in the embodiment of the present application supports data consistency verification in file operation scenarios such as create operation, write operation, append operation, truncate operation, link operation, rename operation, unlink operation, and read operation, as well as complex combination scenarios, and can effectively verify data inconsistency scenarios such as inconsistent file size, inconsistent file content (wrong file content, old data overwriting new data), and entire file loss, thereby making up for the shortcomings of current file testing tools in the industry.

[0106] In another embodiment, the data content of the data block in the test file is generated by a random seed; the mapping value also includes the value of the random seed; after obtaining the expected verification data of any data block, it also includes:

[0107] When the actual verification data does not conform to the expected verification data, the value of the random seed contained in the corresponding mapping value is queried from the hash mapping table according to the key value of any data block; based on the value of the random seed, the data content of any data block is restored, and the data consistency check result of the file after any operation is determined to be verification failure.

[0108] In the specific implementation, if the mapping value of the hash mapping table includes the expected verification data and the value of the random seed, the random seed value of the data block can be obtained from the hash mapping table, and based on the value of the random seed, the data originally written to the data block can be played back, and the original written data and the actual data in the data block can be exported to the local for storage for subsequent analysis and comparison to locate the root cause of the problem that causes data inconsistency.

[0109] In another embodiment, when the file operation is a link operation, the hash mapping table update operation is not to update the hash mapping table; the file after the operation is a link file created based on the test file, and the link file and each data block of the test file have the same data content and index identifier in a one-to-one correspondence; a name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, including: adding the file name of the link file to the test file name list, and saving the number of data blocks and the size of the data blocks included in the link file to the test file name list to obtain an updated name list.

[0110] The link operation is used to create a link file for the test file, such as a hard link. A hard link refers to one or more file names of a file, that is, multiple file names are linked to the same file. After creating a link file for the test file as the source file, the content of the link file is the same as the test file as the source file, that is, the link file has the same data content and index identifiers as each data block of the test file.

[0111] When the user calls a link operation, he inputs the number or proportion of source files to be linked, and the terminal can randomly select the specified number or proportion of test files as source files to perform the link operation. It should be noted that, since at least one type of file operation can be performed on the test file, the test file selected here may be the initially created test file, or it may be a link file after the link operation is performed, or a renamed file after the rename operation is performed. The post-operation file is generated when all file operations of at least one type of operation are completed. For example, assuming that there are 1,000 test files in the current test directory, and the proportion of files to be linked is specified to be 50%, 500 files will be randomly selected to perform the link operation, and the total number of files after the link operation becomes 1,500.

[0112] The file name of the link file is composed of a file prefix name and a file suffix name of the link file, and the file prefix name of the link file is consistent with the file prefix name of the test file.

[0113] To facilitate the understanding of those skilled in the art, Figure 6 A schematic diagram of a test file name list supporting link operations is provided exemplarily. The test file name list includes a test file with a file prefix name of File1 and a test file with a file prefix name of File2, File2-link1 is the first link file created based on the File2 test file, and File2-link2 is the second link file created based on the File2 test file.

[0114] In the specific implementation, when any test file is used as a source file and a link file is created for the source file, the link file is named as "file prefix name + random string", the link file and the source file have the same file prefix name, the file suffix name is the random string, and the file verification data is stored using "file prefix name + data block index identifier", so the source file and all the link files derived from it share a copy of the verification data, so when creating a link file, there is no need to perform any operation on the expected verification data stored in the hash map. That is, the hash map update operation is to not update the hash map.

[0115] However, since the link operation creates a new file, the name of the linked file needs to be added to the test file name list, that is, the number of data blocks and the size of the data blocks included in the linked file need to be saved in the test file name list, and the number of data blocks and the size of the data blocks of the linked file need to be set to be consistent with those of the source file. To improve efficiency, the number of data blocks and the size of data blocks can be encapsulated into an object separately. The source file and all its linked files reference this object, and the reference count variable ref_count is set to represent the total number of source files and linked files. Each time a link operation is performed, ref_count is increased by 1, and each time an unlink operation is performed, ref_count is reduced by 1.

[0116] At least one type of file operation performed on the test file may include a link operation and an unlink operation. After each file operation is performed, a name list update operation needs to be performed, and finally an updated name list is obtained. Among them, the unlink operation is used to delete a linked file. When the user calls the unlink operation, the number or proportion of files to be unlinked is entered, and the terminal randomly selects the specified number or proportion of files to perform the unlink operation.

[0117] After performing the unlink operation on a linked file, you first need to search in the test file name list according to the file name of the linked file, and after finding the linked file, reduce the ref_count it references by 1, and then delete the file name of the linked file from the test file name list. In addition, if ref_count becomes 0 after being reduced by 1, you also need to delete the object containing ref_count, and also delete the verification data of the test file from the hash map. The specific method of deleting the verification data can be to construct a key value based on the file prefix name of the linked file and the data block index identification range, and then find the corresponding key value in the hash map and delete its corresponding mapping value.

[0118] In another embodiment, when the file operation is a renaming operation, the hash mapping table update operation is not to update the hash mapping table; the post-operation file is to modify the file prefix name of the test file to obtain a renamed file with a new file name, and the renamed file has a one-to-one correspondence with each data block of the test file and has the same data content and index identifier; a name list update operation matching the operation type is performed on the test file name list to obtain an updated name list, including: modifying the file prefix name of the test file in the test file name list to the new file name to obtain an updated name list.

[0119] The rename operation is used to rename files. When the user calls the rename operation, he enters the number or ratio of files to be renamed, and the terminal randomly selects the specified number or ratio of test files to perform the rename operation.

[0120] The new file name is composed of a file prefix name and a file suffix name of the renamed file, and the file prefix name of the renamed file is consistent with the file prefix name of the test file.

[0121] A rename operation is performed on a test file. The new file name is named as file prefix name + file suffix name. The file prefix name of the renamed file can be the same as the file prefix name of the test file as the source file, and the file suffix name can be a random string. In addition, each data block of the renamed file and the test file has the same data content and index identifier. Therefore, the verification data of the renamed file and the source file are still the same, so the rename operation does not need to do anything with the verification data, that is, the hash map update operation is not to update the hash map. However, since the rename operation changes the file name, the corresponding file name in the test file name list needs to be modified. The other fields do not need to be modified, and there is no increase or decrease in files involved.

[0122] Since the traditional file data verification method cannot effectively perform data consistency verification on link operation and rename operation scenarios, the above embodiment can support data consistency verification on link operation and rename operation scenarios, thereby improving the accuracy of file data consistency verification.

[0123] In another embodiment, when the file operation is a truncation operation, the post-operation file includes a truncated file obtained after truncating several data blocks of the test file; a hash mapping table update operation matching the operation type is performed on the hash mapping table to obtain an updated hash mapping table, including: generating a key value to be deleted based on the file prefix name of the test file and the index identifier of the data block truncated in the test file; deleting the key value to be deleted and the corresponding mapping value from the hash mapping table to obtain an updated hash mapping table.

[0124] The truncate operation is used to truncate a file, for example, from 2 megabytes to 1 megabyte, truncating the rear part of the file and retaining the front part of the file.

[0125] In a specific implementation, when a user calls a truncation operation, he or she inputs the number or proportion of test files to be truncate, and the terminal can randomly select the specified number or proportion of test files to perform the truncation operation, as well as the truncation proportion for a single test file.

[0126] It should be noted that in order to effectively verify the data consistency after the truncation operation, the truncation operation can be performed for each data block in the test file, because the data consistency check is also based on the data block granularity. Optionally, the truncation operation can truncate N data blocks at the end of the test file. In the extreme case, the entire test file is truncated to obtain an empty file. For example, when the test file is created, the data block size is 4 kilobytes, and the size of the test file is 1 megabyte, then the test file has 256 data blocks; if the truncation operation is selected, the ratio is 30%, then 256*0.3=76 (rounded) data blocks will be truncated, and the test file after the truncation operation will have 180 data blocks remaining, and the file size of the test file becomes 720 kilobytes.

[0127] When performing a truncation operation on a test file, you first need to update the verification data, that is, delete the verification data of the truncated data block. The specific method is to construct the key value to be deleted based on the file prefix name of the test file and the range of the truncated data block index identifier, and find the key value to be deleted and the corresponding mapping value in the hash mapping table for deletion.

[0128] If the test file name list only includes the file names of the test files, when the file operation is a truncation operation, the name list update operation that matches the operation type is not to update the test file name list; if the test file name list also includes the number of data blocks and the size of the data blocks included in each test file, when the file operation is a truncation operation, the name list update operation that matches the operation type includes updating the number of data blocks recorded in the test file name list. As an example, in order to improve the update efficiency, the test file can be searched by the file prefix name. Specifically, the name list update operation performed on the test file name list may include: finding the first test file with the same file prefix name as the test file that has been truncation-operated in the test file name list, and modifying the number of data blocks referenced by it to the value of the number of data blocks remaining after the truncation operation is performed, because the link file and the renamed file created based on the first test file can share the same number of data blocks, so the first test file with the same file prefix name can be found and modified, so that the number of data blocks of the link file and the renamed file with the same file prefix name created based on the first test file will also be automatically modified accordingly.

[0129] In another embodiment, the data content of the data block in the test file is generated by a random seed; when the file operation is an extension operation, the post-operation file includes an extended file obtained after inserting several data blocks into the test file; a hash mapping table update operation matching the operation type is performed on the hash mapping table to obtain an updated hash mapping table, including: calculating the verification data of the data block inserted in the test file as the expected verification data of the inserted data block, and obtaining the value of the random seed corresponding to the inserted data block; determining the mapping value of the inserted data block according to the expected verification data corresponding to the inserted data block and the value of the random seed; generating the key value of the inserted data block according to the file prefix name of the test file and the index identifier of the inserted data block in the extension file; inserting the mapping value of the inserted data block and the key value of the inserted data block into the hash mapping table to obtain an updated hash mapping table.

[0130] The extension operation is used to extend a file, for example, from 1 megabyte to 2 megabytes. For the extension scenario, zeros may be added to the end of the file to extend the file to a specified size.

[0131] In a specific implementation, when a user calls an extended operation, he inputs the number or proportion of test files on which the extended operation is to be performed, and the terminal can randomly select the specified number or proportion of test files to perform the extended operation, as well as the extended proportion for a single test file.

[0132] Optionally, the extension operation can be to extend N data blocks at the end of the test file, and the extended data blocks are all filled with 0. For example, when the test file is created, the data block size is 4 kilobytes, and the size of the test file is 1 megabyte, then the test file has 256 data blocks; if the extension operation is selected, the ratio is 30%, then 76 blocks will be extended, and the number of data blocks of the test file after the extension operation becomes 332, and the file size of the test file becomes 1328 kilobytes.

[0133] When performing an extension operation on a test file, it is necessary to calculate the expected verification data based on the extended data blocks and update the mapping value in the hash mapping table. Since the extended data blocks are all 0 data, the hash mapping table update operation may include: calculating the expected verification data of the extended data blocks, which can be used together with the value of the random seed as the mapping value (the value of the random seed of all 0 data is 0), and then constructing a key value based on the file prefix name of the test file and the index identification range of the extended data block, and inserting the key value and the mapping value into the hash mapping table. If the test file name list only includes the file name of the test file, when the file operation is an extension operation, the name list update operation that matches the operation type is not to update the test file name list; if the test file name list also includes the number of data blocks and the size of the data blocks included in each test file, when the file operation is an extension operation, it is necessary to update the number of data blocks recorded in the test file name list. For details, please refer to the name list update operation corresponding to the above-mentioned truncation operation, which will not be repeated here.

[0134] In another embodiment, at least one type of file operation may also include a write operation for overwriting a file. When the user calls the (write) operation, the number or proportion of the test files to be overwritten is input, and the terminal will randomly select the specified number or proportion of test files to perform the write operation. At the same time, the user also needs to specify the overwrite ratio of a single file, and the overwrite is also based on the granularity of the data block. For example, when creating a test file, the data block size is 4 kilobytes and the file size is 1 megabyte, then the test file has 256 data blocks. If the overwrite ratio of a single test file is specified to be 30%, 256*0.3=76 (rounded) data blocks of the test file will be randomly selected for overwriting.

[0135] Overwriting requires updating the verification data. The method is to calculate the new expected calibration data based on the newly written data for each overwritten data block, and form a new mapping value together with the new random seed value. Then, a key value is constructed based on the file prefix name of the test file and the index identifier of the overwritten data block, and the key value is found in the hash mapping table. The mapping value corresponding to the key value is replaced with the new mapping value. Since overwriting does not change the file size of the test file, there is no need to update the test file name list.

[0136] In another embodiment, at least one type of file operation may also include an append operation. When the user calls the append operation, the user enters the number or proportion of test files to be appended, and the terminal randomly selects the specified number or proportion of files to perform the append operation. At the same time, the user also needs to specify the proportion of appending to a single test file. The appending is also based on the granularity of data blocks. For example, when creating a test file, the data block size is 4 kilobytes and the file size is 1 megabyte, then the file has 256 data blocks. If the append ratio of a single test file is specified to be 30%, 76 data blocks will be appended to the end of the test file. After writing, the number of data blocks of the test file becomes 332, and the size of the test file becomes 1328 kilobytes.

[0137] When appending to a file, the verification data needs to be updated. The method is to calculate the expected verification data based on the appended data block for each newly appended data block, and form a mapping value together with the value of the random seed. Then, the key value is constructed based on the file prefix name of the test file and the index identifier of the appended data block. Finally, the key value and the corresponding mapping value are inserted into the hash mapping table.

[0138] Since appending also changes the file size, the number of data blocks in the test file name list also needs to be updated. The method is to find the first file with the same file prefix name in the test file name list and modify the number of data blocks it references to the value of the number of data blocks after appending.

[0139] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0140] Based on the same inventive concept, the embodiment of the present application also provides a file data verification device for implementing the file data verification method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more file data verification device embodiments provided below can refer to the limitations of the file data verification method above, and will not be repeated here.

[0141] In an exemplary embodiment, Figure 7 As shown, a file data verification device is provided, comprising:

[0142] Acquisition module 710 is used to obtain a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list; the hash mapping table includes key values ​​of data blocks in each test file and corresponding mapping values; the key value is determined based on the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block.

[0143] The operation update module 720 is used to obtain a post-operation file by performing at least one operation type of file operation on the test file, and to perform a name list update operation matching the operation type on the test file name list to obtain an updated name list, and to perform a hash map update operation matching the operation type on the hash map to obtain an updated hash map.

[0144] The data acquisition module 730 is used to calculate the actual verification data of any data block in any of the post-operation files in the updated name list, and query the mapping value corresponding to the key value of any of the data blocks from the updated hash mapping table to obtain the expected verification data of any of the data blocks.

[0145] The verification module 740 is used to determine that the data consistency verification result of any of the operated files is a verification pass when the actual verification data is consistent with the expected verification data.

[0146] In one of the embodiments, the updated name list also includes the number of data blocks and the size of data blocks included in each of the post-operation files; the verification module 740 is specifically used to calculate the actual file size of any of the post-operation files and obtain the expected file size of any of the post-operation files when any of the post-operation files is successfully read; the expected file size is calculated based on the number of data blocks and the size of data blocks included in any of the post-operation files; when the actual file size is consistent with the expected file size, return to the step of calculating the actual verification data of any data block in any of the post-operation files.

[0147] In one of the embodiments, the data content of the data block in the test file is generated by a random seed; the verification module 740 is specifically used to query the value of the random seed contained in the corresponding mapping value from the hash mapping table according to the key value of any of the data blocks when the actual verification data does not conform to the expected verification data; restore the data content of any of the data blocks according to the value of the random seed, and determine that the data consistency verification result of any of the files after the operation is verification failure.

[0148] In one of the embodiments, when the file operation is a link operation, the hash map table update operation is not to update the hash map; the file after the operation is a link file created based on the test file, and the link file has the same data content and index identifier in a one-to-one correspondence with each data block of the test file;

[0149] In one embodiment, the operation update module 720 is specifically used to add the file name of the link file to the test file name list, and save the number of data blocks and the size of the data blocks included in the link file to the test file name list to obtain the updated name list; wherein, the file name of the link file is composed of the file prefix name and the file suffix name of the link file, and the file prefix name of the link file is consistent with the file prefix name of the test file.

[0150] In one of the embodiments, when the file operation is a renaming operation, the hash mapping table update operation is not to update the hash mapping table; the post-operation file is to modify the file prefix name of the test file to obtain a renamed file with a new file name, and the renamed file has the same data content and index identifier in a one-to-one correspondence with each data block of the test file; the operation update module 720 is specifically used to modify the file prefix name of the test file in the test file name list to the new file name to obtain the updated name list; wherein, the new file name is composed of the file prefix name and the file suffix name of the renamed file, and the file prefix name of the renamed file is consistent with the file prefix name of the test file.

[0151] In one of the embodiments, when the file operation is a truncation operation, the post-operation file includes a truncated file obtained after truncating several data blocks of the test file; the operation update module 720 is specifically used to generate a key value to be deleted based on the file prefix name of the test file and the index identifier of the data block truncated in the test file; delete the key value to be deleted and the corresponding mapping value from the hash mapping table to obtain the updated hash mapping table.

[0152] In one of the embodiments, the data content of the data block in the test file is generated by a random seed; when the file operation is an extension operation, the post-operation file includes an extended file obtained after inserting several data blocks into the test file; the operation update module 720 is specifically used to calculate the verification data of the data block inserted in the test file as the expected verification data of the inserted data block, and obtain the value of the random seed corresponding to the inserted data block; determine the mapping value of the inserted data block according to the expected verification data corresponding to the inserted data block and the value of the random seed; generate the key value of the inserted data block according to the file prefix name of the test file and the index identifier of the inserted data block in the extended file; insert the mapping value of the inserted data block and the key value of the inserted data block into the hash mapping table to obtain the updated hash mapping table.

[0153] Each module in the above-mentioned file data verification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0154] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for verifying file data is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0155] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0156] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0157] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0158] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0160] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for verifying file data, characterized in that: The method comprises: Obtain a list of test file names corresponding to each test file and a hash mapping table corresponding to the list of test file names; the hash mapping table includes key values ​​of data blocks in each of the test files and corresponding mapping values; the key values ​​are determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block; Performing at least one type of file operation on the test file to obtain an operated file, performing a name list update operation matching the operation type on the test file name list to obtain an updated name list, and performing a hash map update operation matching the operation type on the hash map to obtain an updated hash map; For any of the post-operation files in the updated name list, calculate actual verification data of any data block in any of the post-operation files, and query a mapping value corresponding to a key value of any of the data blocks from the updated hash mapping table to obtain expected verification data of any of the data blocks; When the actual verification data is consistent with the expected verification data, it is determined that the data consistency verification result of any of the operated files is a verification pass.

2. The method according to claim 1, characterized in that The updated name list also includes the number and size of data blocks included in each of the operated files; Before calculating the actual check data of any data block in any of the post-operation files, the method further includes: In the case of successfully reading any of the post-operation files, calculating the actual file size of any of the post-operation files, and obtaining the expected file size of any of the post-operation files; the expected file size is calculated based on the number of data blocks and the data block size included in any of the post-operation files; When the actual file size is consistent with the expected file size, return to the step of calculating the actual check data of any data block in any of the operated files.

3. The method according to claim 1, characterized in that The data content of the data block in the test file is generated by a random seed; the mapping value also includes the value of the random seed; after obtaining the expected verification data of any of the data blocks, the method further includes: In the case where the actual verification data does not conform to the expected verification data, querying the value of the random seed contained in the corresponding mapping value from the hash mapping table according to the key value of any of the data blocks; According to the value of the random seed, the data content of any of the data blocks is restored, and a data consistency check result of any of the files after the operation is determined to be a check failure.

4. The method according to claim 1, characterized in that In the case where the file operation is a link operation, the hash map table update operation is not to update the hash map; the file after the operation is a link file created based on the test file, and the link file has the same data content and index identifier in a one-to-one correspondence with each data block of the test file; The step of performing a name list update operation matching the operation type on the test file name list to obtain an updated name list includes: Add the file name of the link file to the test file name list, and save the number of data blocks and the size of the data blocks included in the link file to the test file name list to obtain the updated name list; The file name of the link file is composed of a file prefix name and a file suffix name of the link file, and the file prefix name of the link file is consistent with the file prefix name of the test file.

5. The method according to claim 1, characterized in that In the case where the file operation is a renaming operation, the hash mapping table update operation is not to update the hash mapping table; the post-operation file is a file prefix name of the test file modified to obtain a renamed file with a new file name, and the renamed file has the same data content and index identifier as each data block of the test file in a one-to-one correspondence; The step of performing a name list update operation matching the operation type on the test file name list to obtain an updated name list includes: Modify the file prefix names of the test files in the test file name list to the new file names to obtain the updated name list; The new file name is composed of a file prefix name and a file suffix name of the renamed file, and the file prefix name of the renamed file is consistent with the file prefix name of the test file.

6. The method according to claim 1, characterized in that In the case where the file operation is a truncation operation, the post-operation file includes a truncated file obtained by truncating a number of data blocks of the test file; and performing a hash mapping table update operation matching the operation type on the hash mapping table to obtain an updated hash mapping table includes: Generate a key value to be deleted according to the file prefix name of the test file and the index identifier of the data block truncated in the test file; The key value to be deleted and the mapping value corresponding thereto are deleted from the hash mapping table to obtain the updated hash mapping table.

7. The method according to claim 1, characterized in that The data content of the data block in the test file is generated by a random seed; when the file operation is an extension operation, the post-operation file includes an extended file obtained after inserting a plurality of data blocks into the test file; the hash mapping table update operation matching the operation type is performed on the hash mapping table to obtain an updated hash mapping table, including: Calculating the check data of the data block inserted in the test file as the expected check data of the inserted data block, and obtaining the value of the random seed corresponding to the inserted data block; Determining a mapping value of the inserted data block according to expected check data corresponding to the inserted data block and a value of a random seed; Generate a key value of the inserted data block according to the file prefix name of the test file and the index identifier of the inserted data block in the extended file; Inserting the mapping value of the inserted data block and the key value of the inserted data block into the hash mapping table to obtain the updated hash mapping table.

8. A file data verification device, characterized in that: The device comprises: An acquisition module, used to acquire a test file name list corresponding to each test file and a hash mapping table corresponding to the test file name list; the hash mapping table includes key values ​​of data blocks in each test file and corresponding mapping values; the key value is determined according to the file prefix name of the test file to which the data block belongs and the index identifier of the data block in the test file to which it belongs; the mapping value includes expected verification data of the data block; an operation update module, configured to obtain an operated file by performing at least one operation type of file operation on the test file, and to obtain an updated name list by performing a name list update operation matching the operation type on the test file name list, and to obtain an updated hash map by performing a hash map update operation matching the operation type on the hash map; A data acquisition module, for calculating actual verification data of any data block in any of the post-operation files in the updated name list, and querying a mapping value corresponding to a key value of any of the data blocks from the updated hash mapping table to obtain expected verification data of any of the data blocks; The verification module is used to determine that the data consistency verification result of any of the files after the operation is a verification pass when the actual verification data is consistent with the expected verification data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images