A data verification method and device, a storage medium and an electronic device

Abstract

The application provides a data verification method and device, a storage medium and an electronic device, which comprises the following steps: splitting a data file into multiple data blocks, and generating and combining verification codes layer by layer to construct a multi-layer verification structure. First, a first verification code is generated for each data block to form a first verification layer, then the verification codes of the first verification layer are combined to generate a second verification layer, and so on, until a single file verification code is generated to form the uppermost verification layer. This layered and block-based verification strategy not only realizes fine management of data, but also significantly reduces the verification workload through targeted verification, greatly improving the efficiency and accuracy of data verification.

Description

Technical Field

[0001] This invention relates to the field of distributed systems technology, and in particular to a data verification method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the field of distributed systems, with the continuous growth of data volume and the increasing complexity of systems, ensuring data consistency and reliability has become a core challenge. To address the single point of failure that may result from node failures, distributed systems typically employ data redundancy strategies, storing multiple copies of data across multiple nodes. While this redundant storage mechanism enhances data availability and fault tolerance, it also introduces new challenges: the need to periodically verify the consistency of these copies and to perform rapid and effective repairs when data corruption or loss is detected.

[0003] However, since the amount of data in distributed systems is usually very large, traditional data verification and repair methods are often inefficient and cannot meet the requirements of real-time or near real-time.

[0004] Therefore, how to efficiently verify data files has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a data verification method, apparatus, storage medium, and electronic device that overcomes or at least partially solves the above problems, the technical solution of which is as follows:

[0006] A data validation method, comprising:

[0007] Obtain the data file;

[0008] The data file is split into multiple data blocks, and a first check code is generated for each data block. The first check code of each data block and its corresponding first data range constitute the first check layer of the data file.

[0009] The first check codes in the first check layer are combined and verified to generate multiple second check codes. The second check layer of the data file is composed of each second check code and its corresponding second data range.

[0010] The second check codes in the second verification layer are combined and verified to generate multiple third check codes. The third check code and its corresponding third data range constitute the third verification layer of the data file.

[0011] The combined verification process continues until a single file verification code for the data file is generated. The file verification code and its corresponding file data range constitute the topmost verification layer of the data file.

[0012] Optionally, after generating a single file checksum for the data file, the method further includes:

[0013] Determine whether the file check codes of the primary data file and the backup data file are the same. If not, start from the next check layer after the top check layer and compare the check codes of the primary data file and the backup data file in each check layer in turn until a first target check code that is inconsistent between the primary data file and the backup data file is found in each of the first check codes in the first check layer.

[0014] The first target data block corresponding to the first target check code is located in the primary data file and the backup data file, respectively.

[0015] Optionally, after locating the first target data block corresponding to the first target checksum in the primary data file and the backup data file respectively, the method further includes:

[0016] The first target data block in the primary data file is used to repair the data in the backup data file.

[0017] Optionally, after generating a single file checksum for the data file, the method further includes:

[0018] Obtain the range of data to be verified;

[0019] In each verification layer of the primary data file and the backup data file, a target data range containing the range of data to be verified and having the smallest range is determined.

[0020] Within the target data range, locate the second target data block corresponding to the data range to be verified in the primary data file and the backup data file, respectively.

[0021] By comparing the first checksum of the second target data block in the primary data file and the backup data file, it is determined that there is an inconsistent second target checksum between the primary data file and the backup data file;

[0022] The third target data block corresponding to the second target check code is located in both the primary data file and the backup data file.

[0023] Optionally, after locating the third target data block corresponding to the second target checksum in the primary data file and the backup data file respectively, the method further includes:

[0024] The third target data block in the primary data file is used to repair the data in the backup data file.

[0025] Optionally, after generating a single file checksum for the data file, the method further includes:

[0026] When at least one fourth target data block is added to the data file, the first check code is generated for each fourth target data block, and the first check code of the fourth target data block and its corresponding first data range are added to the first verification layer to update the first verification layer.

[0027] Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file.

[0028] Perform combined verification on each of the second check codes in the updated second verification layer, and update the third verification layer of the data file;

[0029] The combined verification process continues until the file verification code of the data file is updated.

[0030] Optionally, after generating a single file checksum for the data file, the method further includes:

[0031] In the event that at least one fifth target data block is deleted from the data file, the first checksum of the fifth target data block and its corresponding first data range are deleted from the first verification layer of the data file, and the first data range corresponding to each first checksum in the first verification layer is redefined to update the first verification layer.

[0032] Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file.

[0033] Perform combined verification on each of the second check codes in the updated second verification layer, and update the third verification layer of the data file;

[0034] The combined verification process continues until the file verification code of the data file is updated.

[0035] A data verification device includes: a data file acquisition unit, a first verification code generation unit, a second verification code generation unit, a third verification code generation unit, and a file verification code generation unit.

[0036] The data file acquisition unit is used to acquire a data file;

[0037] The first check code generation unit is used to split the data file into multiple data blocks, generate a first check code for each data block, and the first check code of each data block and its corresponding first data range constitute the first check layer of the data file.

[0038] The second check code generation unit is used to perform combined verification on each of the first check codes in the first verification layer to generate multiple second check codes, and the second verification layer of the data file is composed of each second check code and its corresponding second data range.

[0039] The third check code generation unit is used to perform combined verification on each of the second check codes in the second verification layer to generate multiple third check codes, and the third verification layer of the data file is composed of each of the third check codes and its corresponding third data range.

[0040] The file verification code generation unit is used to continuously perform a combined verification process until a single file verification code for the data file is generated. The file verification code and its corresponding file data range constitute the topmost verification layer of the data file.

[0041] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the data verification method described in any of the preceding claims.

[0042] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the data verification method described above.

[0043] By employing the above technical solution, this invention provides a data verification method, apparatus, storage medium, and electronic device, which obtains a data file; splits the data file into multiple data blocks, generates a first checksum for each data block, and the first checksum of each data block and its corresponding first data range constitute the first verification layer of the data file; combines and verifies each first checksum in the first verification layer to generate multiple second checksums, and each second checksum and its corresponding second data range constitute the second verification layer of the data file; combines and verifies each second checksum in the second verification layer to generate multiple third checksums, and each third checksum and its corresponding third data range constitute the third verification layer of the data file; the combination and verification process continues until a single file checksum is generated for the data file, and the file checksum and its corresponding file data range constitute the top-level verification layer of the data file. This invention adopts a hierarchical and block-based management strategy to achieve fine-grained organization of data, enabling targeted verification of inconsistent data blocks, thereby reducing the workload of data verification and improving data verification efficiency.

[0044] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0045] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0046] Figure 1 A flowchart illustrating one embodiment of the data verification method provided by this invention is shown.

[0047] Figure 2 This diagram illustrates a tree-structured hierarchical data structure provided by an embodiment of the present invention.

[0048] Figure 3 A flowchart illustrating another implementation of the data verification method provided in this invention is shown.

[0049] Figure 4 A flowchart illustrating another implementation of the data verification method provided in this invention is shown.

[0050] Figure 5 A flowchart illustrating another implementation of the data verification method provided in this invention is shown.

[0051] Figure 6 This illustration shows another tree-structured hierarchical data provided by an embodiment of the present invention;

[0052] Figure 7 A flowchart illustrating another implementation of the data verification method provided in this invention is shown.

[0053] Figure 8 This illustration shows another tree-structured hierarchical data provided by an embodiment of the present invention;

[0054] Figure 9 A schematic diagram of the data verification device provided in an embodiment of the present invention is shown;

[0055] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0056] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0057] In the context of today's rapid development of information technology, distributed systems have become a key technological architecture for processing large-scale data and providing highly available services. With the widespread application of technologies such as cloud computing, big data, and the Internet of Things, the amount of data in distributed systems is experiencing explosive growth, and system complexity is also increasing accordingly. In this environment, ensuring data consistency and reliability is not only a technical challenge, but also the cornerstone of business continuity and user trust.

[0058] To address single points of failure caused by node failures, network partitions, or other unexpected events, distributed systems commonly employ data redundancy strategies. This strategy improves data availability and fault tolerance by storing multiple copies of data across multiple nodes. However, data redundancy also introduces a series of new management challenges, the most critical of which is how to effectively verify data consistency and quickly repair corrupted or lost data.

[0059] Traditional data verification and repair methods, such as integrity checks and manual repair, often fall short when dealing with massive amounts of data due to their linear processing approach and lack of intelligent optimization. These methods are not only time-consuming, but also easily become performance bottlenecks in large-scale distributed systems, failing to meet the real-time or near-real-time data processing needs of modern applications.

[0060] Based on this, this invention provides a data verification method that constructs a multi-layered verification structure by splitting a data file into multiple data blocks and generating and combining check codes layer by layer. The method first generates a first check code for each data block, forming a first verification layer. Then, the check codes of the first verification layer are combined to generate a second verification layer, and so on, until a single file check code is generated, constituting the top-level verification layer. This layered and block-based verification strategy not only achieves refined data management but also significantly reduces the workload of verification through targeted verification, greatly improving the efficiency and accuracy of data verification.

[0061] like Figure 1 The diagram shows a flowchart of one embodiment of the data verification method provided by this invention. The method may include:

[0062] S100, Obtain the data file.

[0063] A data file refers to a physical or logical entity that stores data. It can contain various forms of information, such as text, numbers, images, audio, and video. Data files are typically stored in a specific format so that computer systems can read and process this information.

[0064] Specifically, embodiments of the present invention can obtain data files stored on any local computing node in a distributed architecture, or can obtain data files received from other computing nodes through a network interface.

[0065] S110. The data file is split into multiple data blocks, and a first check code is generated for each data block. The first check code of each data block and its corresponding first data range constitute the first check layer of the data file.

[0066] In this context, a data block refers to a smaller, fixed-size portion of a data file. The size of a data block can be set according to the specific application requirements.

[0067] Optionally, embodiments of the present invention can split the data file into multiple data blocks of the same size according to a fixed block size, to adapt to most distributed file systems and cloud storage services. For example, the size of each data block is 1MB.

[0068] Optionally, embodiments of the present invention can also determine the size of each data block based on the data content. For example, based on the file type of the data file, in the case of an image or video file, the size of each data block is adjusted according to the compression format and resolution of the data file in order to better utilize the characteristics of the decoder.

[0069] In this context, the checksum refers to the code used to verify the integrity of the data block content. The purpose of the checksum is to allow the receiver or reader to check whether the data block has been erroneous or corrupted during transmission or storage by recalculating the checksum and comparing it with the checksum provided by the sender.

[0070] Specifically, in embodiments of the present invention, a specified hash algorithm can be applied to the data block to map the data content in the data block to a fixed-length string, i.e., the checksum of the data block.

[0071] Optionally, the hash algorithm may include CRC32 (Cyclic Redundancy Check 32-bit) algorithm, MD5 (Message-Digest algorithm 5) algorithm, SHA-1 (Secure Hash Algorithm 1) algorithm, SHA-256 (Secure Hash Algorithm 256-bit) algorithm and SHA-3 (Secure Hash Algorithm 3) algorithm.

[0072] In this context, the data range refers to the starting position and length of a data block within a data file. The data range can be represented by a byte offset and the size of the data block. For example, if a data file is divided into multiple 1MB data blocks, the data range of the first data block might be from 0 bytes to 1MB-1 bytes, the data range of the second data block might be from 1MB bytes to 2MB-1 bytes, and so on.

[0073] S120. Combine and verify each first check code in the first verification layer to generate multiple second check codes. The second verification layer of the data file is composed of each second check code and its corresponding second data range.

[0074] Specifically, in this embodiment of the invention, multiple consecutive first check codes within a first data range are combined in a first verification layer according to a specified number of first check code combinations to obtain a first combined check code. A specified hash algorithm is used to generate a corresponding second check code from the first combined check code, and the data range formed by the consecutive first data ranges is used as the second data range corresponding to the second check code.

[0075] The number of first check code combinations can be determined based on the total number of data blocks, and the ratio of the total number to the number of check code combinations should be an integer as much as possible.

[0076] It should be noted that, in this embodiment of the invention, the same hash algorithm or different hash algorithms can be used for checksum calculation in each checksum layer. If different hash algorithms are used, a checksum calculation strategy can be pre-defined, in which the hash algorithm used for checksum calculation in each checksum layer is determined.

[0077] S130. Perform combined verification on each of the second check codes in the second verification layer to generate multiple third check codes. The third check code and its corresponding third data range constitute the third verification layer of the data file.

[0078] Specifically, in this embodiment of the invention, multiple consecutive second check codes within a second data range are combined in the second verification layer according to a specified number of second check code combinations to obtain a second combined check code. A specified hash algorithm is used to generate a corresponding third check code from the second combined check code, and the data range consisting of consecutive second data ranges is taken as the third data range corresponding to the third check code.

[0079] The number of second check code combinations can be the same as the number of first check code combinations, or it can be determined based on the total number of second check codes in the second check, so that the ratio of the total number to the number of second check code combinations is an integer as much as possible.

[0080] S140. Continue the combined verification process until a single file check code is generated for the data file. The file check code and its corresponding file data range constitute the top-level verification layer of the data file.

[0081] This invention first divides the data file into N equal-sized data blocks, each generating an independent checksum. These checksums and their corresponding data ranges constitute the first verification layer (the bottom layer). Next, the checksums of multiple data blocks in the first verification layer are combined and verified to generate new checksums and their corresponding data ranges, forming the second verification layer. Then, the checksums of the second verification layer are further combined and verified to form the third verification layer. This process continues, with each layer combining multiple lower-level checksums into a new checksum and recording its corresponding data range, until a tree structure composed of multiple verification layers is formed. Finally, after multiple layers of verification, the topmost verification layer will contain only one checksum, which represents the integrity of the entire file.

[0082] For example, using a specified hash algorithm, every 16 checksums are combined into a new checksum. For a 1GB data file, if it is divided into 1MB segments (1024 data blocks), each data block is individually checked to form a first checksum. Then, every 16 first checksums are used to calculate a second checksum, resulting in 64 second checksums. Each second checksum corresponds to a data range; for example, the first second checksum in the second layer has a range of [0, 16x1024], representing the checksums of the previous 16 data blocks. Next, every 16 second checksums in the second layer are used to calculate a third checksum, resulting in 4 third checksums in the third layer. Finally, the 4 third checksums in the third layer are used to calculate a fourth checksum, which is the file's checksum. This hierarchical verification method not only ensures the integrity of the data file but also provides an efficient data verification mechanism through its tree structure.

[0083] For ease of understanding, this section combines... Figure 2 Explanation: Figure 2 The diagram illustrates a tree-structured hierarchical data structure according to an embodiment of the present invention. Assume a 9MB data file is divided into nine 1MB data blocks. Each data block generates a checksum, forming the first verification layer with nine checksums, each corresponding to a specific data range. Next, the checksums in the first verification layer are combined, with every three checksums generating a new checksum. These new checksums form the second verification layer. The second verification layer has three checksums, each corresponding to a data range. For example, the first checksum of the second verification layer is calculated by combining the first three checksums from the first verification layer, with a data range of [0, 3MB]. Finally, the checksums in the second verification layer are further combined, with every three checksums generating a new checksum. Since the second verification layer has only three checksums, the third verification layer has only one checksum, which represents the integrity of the entire 9MB data file and is the final file checksum. This hierarchical verification method, by progressively combining checksums, forms a tree structure composed of three layers of checksums, ensuring the integrity and correctness of the data file.

[0084] This invention provides a data verification method that involves obtaining a data file; dividing the data file into multiple data blocks; generating a first checksum for each data block; and forming a first verification layer for the data file using the first checksums of each data block and their corresponding first data ranges. The first checksums in the first verification layer are then combined and verified to generate multiple second checksums, which in turn form a second verification layer for the data file. The second checksums in the second verification layer are then combined and verified to generate multiple third checksums, which in turn form a third verification layer for the data file. This combined verification process continues until a single file checksum is generated, which, along with its corresponding file data range, forms the topmost verification layer of the data file. This invention employs a hierarchical and block-based management strategy, enabling refined data organization and targeted verification of inconsistent data blocks, thereby reducing the workload of data verification and improving its efficiency.

[0085] In distributed systems, the primary data file and backup data file are crucial components for ensuring data reliability and high system availability. The primary data file is typically managed by the primary node or primary server. The backup data file is a copy of the primary data file, usually storing the same data. In distributed systems, backup data files are typically maintained by backup nodes or backup servers, which periodically or in real-time synchronize data from the primary node to ensure consistency between the two. Therefore, the consistency between the primary and backup data files needs to be verified during the storage of backup data files.

[0086] Optional, based on Figure 1 The method shown is as follows: Figure 3 As shown in the flowchart of another embodiment of the data verification method provided by the present invention, after generating a single file verification code for the data file, the data verification method may further include:

[0087] S300. Determine whether the file checksums of the primary data file and the backup data file are the same. If not, proceed to step S310.

[0088] This invention allows for a quick check of the overall consistency between the primary and backup data files by comparing their file checksums. If the checksums are identical, it confirms that the data content is consistent, and the backup data file was not damaged or lost during transmission and storage. If they differ, it indicates that at least a portion of the data between the primary and backup data files is different, requiring further detailed examination of the checksums at each check layer.

[0089] S310. Starting from the next verification layer after the top verification layer, compare the check codes of the primary data file and the backup data file in each verification layer in turn, until a first target check code that is inconsistent between the primary data file and the backup data file is found in each first check code of the first verification layer.

[0090] In this embodiment of the invention, starting from the layer below the topmost verification layer, the verification codes of the primary data file and the backup data file are compared layer by layer until a first target verification code that is inconsistent between the primary data file and the backup data file is found in the first verification layer.

[0091] S320. Locate the first target data block corresponding to the first target check code in the primary data file and the backup data file respectively.

[0092] Based on the correspondence between check codes and data blocks in the first verification layer, this embodiment of the invention can quickly locate the first target data block corresponding to the first target check code in the primary data file and the backup data file.

[0093] To facilitate understanding, an example is provided below: When performing a full data verification on a data file, if the data file has three verification layers, the checksums of the topmost (third) verification layer of the primary and backup data files are compared first. If the checksums match, the primary and backup data files are considered identical. If they do not match, all checksums in the second verification layer are compared. This identifies the inconsistent checksums in the second verification layer. For these inconsistent checksums in the second verification layer, the checksums in their respective first verification layers are then compared. By comparing the checksums in the first verification layer, the specific inconsistent data block can be accurately located.

[0094] This invention initially determines whether the primary and backup data files are consistent by comparing their checksums. If they are inconsistent, the checksums are compared layer by layer until an inconsistent checksum is found at the first checksum layer. Then, the data block corresponding to the inconsistent checksum is located in both data files. This layer-by-layer verification method avoids comparing all data blocks, greatly improving the efficiency of locating discrepancies.

[0095] Optionally, in the above Figure 3 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, after locating the first target data block corresponding to the first target check code in the primary data file and the backup data file respectively, the method may further include:

[0096] The first target data block in the primary data file is used to repair the data in the first target data block in the backup data file.

[0097] Specifically, in this embodiment of the invention, the contents of the first target data block in the primary data file can be copied to the first target data block in the backup data file, thereby repairing the data in the backup data file and restoring it to a correct state.

[0098] This invention identifies data blocks that differ between the primary and backup data files through full data verification. These data blocks are then repaired individually, which can quickly restore lost or corrupted data in the backup data file and ensure the consistency and reliability of data in the distributed system.

[0099] In distributed systems, to ensure long-term stable operation, regular maintenance and inspections are performed, including local data verification to identify and correct potential data corruption issues.

[0100] Optional, based on Figure 1 The method shown is as follows: Figure 4 As shown in the flowchart of another embodiment of the data verification method provided by the present invention, after generating a single file verification code for the data file, the data verification method may further include:

[0101] S400: Obtain the range of data to be verified.

[0102] Specifically, embodiments of the present invention can obtain the range of data to be verified input by maintenance personnel, and can also identify the range of data to be verified pre-set in a scheduled verification task.

[0103] S410. Determine the target data range that contains the smallest range of data to be verified in each verification layer of the primary data file and the backup data file.

[0104] Specifically, in this embodiment of the invention, the data range in each verification layer can be used as an index to quickly locate the target data range that contains the smallest range of data to be verified in each verification layer.

[0105] S420. Within the target data range, locate the second target data block corresponding to the data range to be verified in the primary data file and the backup data file, respectively.

[0106] According to the embodiments of the present invention, the second target data block corresponding to the data range to be verified can be quickly located in the primary data file and the backup data file based on the correspondence between the data range and the data block.

[0107] S430. Compare the first checksums of the second target data blocks in the primary data file and the backup data file to determine that there are inconsistent second target checksums between the primary data file and the backup data file.

[0108] In this embodiment of the invention, the first checksum of each corresponding second target data block in the primary data file and the backup data file can be compared to filter out the second target checksums that are inconsistent between the primary data file and the backup data file.

[0109] S440. Locate the third target data block corresponding to the second target check code in the primary data file and the backup data file respectively.

[0110] Based on the correspondence between check codes and data blocks in the first verification layer, this embodiment of the invention can quickly locate the third target data block corresponding to the second target check code in the primary data file and the backup data file.

[0111] To facilitate understanding, an example is given here: Based on Figure 2 The tree-structured hierarchical data is shown. Assume the range of data to be verified is [4M, 6M]. In the verification layer of the data file, find the data range that contains the range to be verified. For example, in the second verification layer, find the data range [3M, 6M], which includes the range [4M, 6M]. Then, in the first verification layer of the primary and backup data files, locate the second target data block corresponding to the data range [3M, 6M]. The data range of the 5th and 6th data blocks in the first verification layer is exactly [4M, 6M]. Compare the first checksums of the 5th and 6th data blocks in the primary and backup data files to determine that there is an inconsistent second target checksum between the primary and backup data files. Then, locate the third target data block corresponding to the second target checksum in the primary and backup data files.

[0112] In this embodiment of the invention, the smallest data range containing the data range to be verified is found in each verification layer of the primary and backup data files. Within the smallest target data range, specific data blocks corresponding to the data range to be verified are precisely located in the primary and backup data files. The check codes of these data blocks in the primary and backup data files are then compared to determine whether there is any inconsistency. If inconsistent check codes are found, the specific data blocks corresponding to these inconsistent check codes are further located in the primary and backup data files. Through this layered verification method, the consistency of data files within a specific data range can be compared efficiently and accurately, ensuring the integrity and reliability of the data.

[0113] Optionally, in the above Figure 4 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, after locating the third target data block corresponding to the second target check code in the primary data file and the backup data file respectively, the method may further include:

[0114] The third target data block in the primary data file is used to repair the data in the third target data block in the backup data file.

[0115] Specifically, in this embodiment of the invention, the contents of the third target data block in the primary data file can be copied to the third target data block in the backup data file, thereby repairing the data in the backup data file and restoring it to a correct state.

[0116] This invention identifies data blocks that differ between the primary and backup data files through partial data verification. These data blocks are then repaired individually, which can quickly restore lost or corrupted data in the backup data file and ensure the consistency and reliability of data in the distributed system.

[0117] Optional, based on Figure 1 The method shown is as follows: Figure 5 As shown in the flowchart of another embodiment of the data verification method provided by the present invention, after generating a single file verification code for the data file, the data verification method may further include:

[0118] S500: When at least one fourth target data block is added to the data file, a first check code is generated for each fourth target data block, and the first check code of the fourth target data block and its corresponding first data range are added to the first check layer to update the first check layer.

[0119] In the field of distributed systems, newly added data blocks are typically located at the end of a data file. This embodiment of the invention can record the first checksum and its corresponding first data range of the newly added fourth target data block at the end of the first checksum layer.

[0120] S510. Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file.

[0121] Understandably, the updated first verification layer contains multiple first verification codes, which need to be recombined and verified to generate and update the second verification layer of the data file.

[0122] S520. Perform combined verification on each of the second check codes in the updated second verification layer, and update the third verification layer of the data file.

[0123] Understandably, the updated second verification layer contains multiple second verification codes, which need to be recombined and verified to generate and update the third verification layer of the data file.

[0124] S530: Continue the combined verification process until the file checksum of the data file is updated.

[0125] The embodiments of the present invention can continue to perform the combined verification process, updating layer by layer, until the topmost file verification code of the data file is generated and updated.

[0126] For ease of understanding, this section combines... Figure 6 Explanation: Figure 6 The diagram illustrates another tree-structured hierarchical data structure provided by an embodiment of the present invention. When new data needs to be written to the data file, such as two new data blocks (marked in black), the checksums of these two new data blocks are calculated and added to the end of the first checksum layer. Since every three first checksums form a second checksum, the third checksum of the second checksum layer (marked in shaded) needs to be updated. This checksum was previously composed of two checksums from the first checksum layer, but now it consists of three, so its checksum and corresponding data range need to be updated. After the update, the second checksum layer will contain four checksums. The first three checksums of the second checksum layer are combined into one checksum, which serves as the first checksum of the third checksum layer. The fourth checksum of the second checksum layer directly serves as the second checksum of the third checksum layer. The two checksums of the third checksum layer are combined into one checksum, which serves as the file checksum of the fourth checksum layer. Through this bottom-up, layer-by-layer update method, even after new data is added, it can still be ensured that the checksums of the entire tree structure are up-to-date, thereby guaranteeing the integrity and consistency of the data file.

[0127] In this embodiment of the invention, when a new data block is added to a data file, a first checksum is generated for each new data block and added to a first checksum layer. The checksums in each checksum layer are then updated layer by layer until the file checksum of the data file is updated. By generating and updating checksums layer by layer, this embodiment of the invention ensures the integrity and consistency of each data block in the data file.

[0128] Optional, based on Figure 1 The method shown is as follows: Figure 7 As shown in the flowchart of another embodiment of the data verification method provided by the present invention, after generating a single file verification code for the data file, the data verification method may further include:

[0129] S700: If at least one fifth target data block is deleted from the data file, delete the first checksum of the fifth target data block and its corresponding first data range in the first checksum of the data file, and redetermine the first data range corresponding to each first checksum in the first checksum of the first checksum to update the first checksum.

[0130] When at least one fifth target data block is deleted from the data file, the first checksum of these data blocks and their corresponding first data ranges need to be deleted in the first verification layer. Simultaneously, the first data ranges corresponding to the remaining first checksums in the first verification layer are redefined to update the first verification layer.

[0131] S710. Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file.

[0132] Understandably, the updated first verification layer contains multiple first verification codes, which need to be recombined and verified to generate and update the second verification layer of the data file.

[0133] S720. Perform combined verification on each of the second check codes in the updated second check layer, and update the third check layer of the data file.

[0134] Understandably, the updated second verification layer contains multiple second verification codes, which need to be recombined and verified to generate and update the third verification layer of the data file.

[0135] S730: Continue the combined verification process until the file checksum of the data file is updated.

[0136] The embodiments of the present invention can continue to perform the combined verification process, updating layer by layer, until the topmost file verification code of the data file is generated and updated.

[0137] For ease of understanding, this section combines... Figure 8 Explanation: Figure 8 The diagram shown is an illustration of another type of tree-structured hierarchical data provided in an embodiment of the present invention, based on... Figure 2 The tree-structured data shown illustrates how, when data is deleted from the data file, for example, the fifth data block (marked by a dashed box), the deleted data block's checksum and data range are removed. Since every three first checksums form a second checksum, the checksums of the second and subsequent checksums in the second checksum layer need to be updated (marked with shaded areas). The data range of the second and subsequent checksums in the second checksum layer is then redefined. The second checksum of the second checksum layer is obtained by combining the fourth to sixth checksums from the original first checksum layer. This process is repeated for each checksum in the second checksum layer, and so on. This bottom-up, layer-by-layer update method ensures that the checksums of the entire tree structure remain up-to-date after data deletion, thus guaranteeing the integrity and consistency of the data file.

[0138] In this embodiment of the invention, when deleting a data block from a data file, the data range corresponding to each checksum in the first verification layer is redefined, and the checksums in the verification layers are updated layer by layer until the file checksum of the data file is updated. By generating and updating checksums layer by layer, this embodiment of the invention ensures the integrity and consistency of each data block in the data file.

[0139] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0140] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0141] Corresponding to the above method embodiments, this invention also provides a data verification device, the structure of which is as follows: Figure 9 As shown, it may include: a data file acquisition unit 10, a first verification code generation unit 20, a second verification code generation unit 30, a third verification code generation unit 40, and a file verification code generation unit 50.

[0142] The data file acquisition unit 10 is used to acquire data files.

[0143] The first checksum generation unit 20 is used to split the data file into multiple data blocks, generate a first checksum for each data block, and the first checksum of each data block and its corresponding first data range constitute the first checksum layer of the data file.

[0144] The second verification code generation unit 30 is used to perform combined verification on each of the first verification codes in the first verification layer to generate multiple second verification codes, and the second verification layer of the data file is composed of each second verification code and its corresponding second data range.

[0145] The third check code generation unit 40 is used to perform combined verification on each of the second check codes in the second verification layer to generate multiple third check codes, and the third check code and its corresponding third data range constitute the third verification layer of the data file.

[0146] The file verification code generation unit 50 is used to continuously perform a combined verification process until a single file verification code for the data file is generated. The file verification code and its corresponding file data range constitute the topmost verification layer of the data file.

[0147] Optionally, the data verification device may further include: a file verification code consistency determination unit, a first target verification code query unit, and a first target data block positioning unit.

[0148] The file checksum consistency determination unit is used to determine whether the file checksums of the primary data file and the backup data file are the same after the file checksum generation unit 50 generates a single file checksum for the data file. If not, the first target checksum query unit is triggered.

[0149] The first target check code query unit is used to compare the check codes of the primary data file and the backup data file in each check layer, starting from the check layer below the top check layer, until a first target check code that is inconsistent between the primary data file and the backup data file is found in each of the first check codes in the first check layer.

[0150] The first target data block positioning unit is used to locate the first target data block corresponding to the first target check code in the primary data file and the backup data file, respectively.

[0151] Optionally, the data verification device may further include: a first data repair unit.

[0152] The first data repair unit is used to repair the first target data block in the backup data file after the first target data block positioning unit locates the first target data block corresponding to the first target check code in the primary data file and the backup data file respectively.

[0153] Optionally, the data verification device may further include: a data range acquisition unit, a target data range determination unit, a second target data block positioning unit, a second target verification code determination unit, and a third target data block positioning unit.

[0154] The unit for obtaining the range of data to be verified is used to obtain the range of data to be verified.

[0155] The target data range determination unit is used to determine the target data range that contains the range of data to be verified and has the smallest range in each verification layer of the primary data file and the backup data file.

[0156] The second target data block positioning unit is used to locate the second target data block corresponding to the range of data to be verified in the primary data file and the backup data file, respectively, within the target data range.

[0157] The second target check code determination unit is used to compare the first check code of the second target data block in the primary data file and the backup data file, and determine that there is an inconsistent second target check code between the primary data file and the backup data file.

[0158] The third target data block positioning unit is used to locate the third target data block corresponding to the second target check code in the primary data file and the backup data file, respectively.

[0159] Optionally, the data verification device may also include a second data repair unit.

[0160] The second data repair unit is used to repair the third target data block in the backup data file after the third target data block positioning unit locates the third target data block corresponding to the second target check code in the primary data file and the backup data file respectively.

[0161] Optionally, the data verification device may also include: a new data update unit.

[0162] A new data update unit is added for use after the file verification code generation unit 50 generates a single file verification code for the data file. When at least one fourth target data block is added to the data file, the unit generates a first verification code for each fourth target data block and adds the first verification code of the fourth target data block and its corresponding first data range to the first verification layer to update the first verification layer. The unit then performs combined verification on each of the first verification codes in the updated first verification layer to update the second verification layer of the data file. Finally, it performs combined verification on each of the second verification codes in the updated second verification layer to update the third verification layer of the data file. This combined verification process continues until the file verification code of the data file is updated.

[0163] Optionally, the data verification device may further include a data deletion and update unit.

[0164] The data deletion update unit is used after the file verification code generation unit 50 generates a single file verification code for the data file. If at least one fifth target data block is deleted from the data file, the unit deletes the first verification code of the fifth target data block and its corresponding first data range from the first verification layer of the data file, and redetermines the first data range corresponding to each first verification code in the first verification layer to update the first verification layer. The unit then performs combined verification on each first verification code in the updated first verification layer to update the second verification layer of the data file. Finally, it performs combined verification on each second verification code in the updated second verification layer to update the third verification layer of the data file. This combined verification process continues until the file verification code of the data file is updated.

[0165] This invention provides a data verification device that obtains a data file; divides the data file into multiple data blocks, generates a first checksum for each data block, and the first checksum of each data block and its corresponding first data range constitute a first verification layer of the data file; combines and verifies the first checksums in the first verification layer to generate multiple second checksums, and the second checksums and their corresponding second data ranges constitute a second verification layer of the data file; combines and verifies the second checksums in the second verification layer to generate multiple third checksums, and the third checksums and their corresponding third data ranges constitute a third verification layer of the data file; the combination and verification process continues until a single file checksum is generated for the data file, and the file checksum and its corresponding file data range constitute the topmost verification layer of the data file. This invention adopts a hierarchical and block-based management strategy to achieve fine-grained organization of data, enabling targeted verification of inconsistent data blocks, thereby reducing the workload of data verification and improving data verification efficiency.

[0166] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0167] The data verification device includes a processor and a memory. The data file acquisition unit 10, the first verification code generation unit 20, the second verification code generation unit 30, the third verification code generation unit 40, and the file verification code generation unit 50 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0168] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, data files can be split into multiple data blocks, and checksums can be generated and combined layer by layer to construct a multi-level checksum structure. This method first generates a first checksum for each data block, forming the first checksum layer. Then, the checksums of the first checksum layer are combined to generate the second checksum layer, and so on, until a single file checksum is generated, forming the top-level checksum layer. This enables fine-grained data management, significantly reduces the workload of checksum verification through targeted verification, and greatly improves the efficiency and accuracy of data verification.

[0169] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the data verification method.

[0170] This invention provides a processor for running a program, wherein the program executes the data verification method during runtime.

[0171] like Figure 10 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned data verification method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0172] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program with initialization steps including a data verification method.

[0173] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0175] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0176] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0178] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0179] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A data verification method, characterized in that, include: Obtain the data file; The data file is split into multiple data blocks, and a first check code is generated for each data block. The first check code of each data block and its corresponding first data range constitute the first check layer of the data file. The first check codes in the first check layer are combined and verified to generate multiple second check codes. The second check layer of the data file is composed of each second check code and its corresponding second data range. The second check codes in the second verification layer are combined and verified to generate multiple third check codes. The third check code and its corresponding third data range constitute the third verification layer of the data file. The combined verification process continues until a single file verification code for the data file is generated. The file verification code and its corresponding file data range constitute the topmost verification layer of the data file.

2. The method according to claim 1, characterized in that, After generating a single file checksum for the data file, the method further includes: Determine whether the file check codes of the primary data file and the backup data file are the same. If not, start from the next check layer after the top check layer and compare the check codes of the primary data file and the backup data file in each check layer in turn until a first target check code that is inconsistent between the primary data file and the backup data file is found in each of the first check codes in the first check layer. The first target data block corresponding to the first target check code is located in the primary data file and the backup data file, respectively.

3. The method according to claim 2, characterized in that, After locating the first target data block corresponding to the first target checksum in the primary data file and the backup data file respectively, the method further includes: The first target data block in the primary data file is used to repair the data in the backup data file.

4. The method according to claim 1, characterized in that, After generating a single file checksum for the data file, the method further includes: Obtain the range of data to be verified; In each verification layer of the primary data file and the backup data file, a target data range containing the range of data to be verified and having the smallest range is determined. Within the target data range, locate the second target data block corresponding to the data range to be verified in the primary data file and the backup data file, respectively. By comparing the first checksum of the second target data block in the primary data file and the backup data file, it is determined that there is an inconsistent second target checksum between the primary data file and the backup data file; The third target data block corresponding to the second target check code is located in both the primary data file and the backup data file.

5. The method according to claim 4, characterized in that, After locating the third target data block corresponding to the second target checksum in the primary data file and the backup data file respectively, the method further includes: The third target data block in the primary data file is used to repair the data in the backup data file.

6. The method according to any one of claims 1 to 5, characterized in that, After generating a single file checksum for the data file, the method further includes: When at least one fourth target data block is added to the data file, the first check code is generated for each fourth target data block, and the first check code of the fourth target data block and its corresponding first data range are added to the first verification layer to update the first verification layer. Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file; Perform combined verification on each of the second check codes in the updated second verification layer, and update the third verification layer of the data file; The combined verification process continues until the file verification code of the data file is updated.

7. The method according to any one of claims 1 to 5, characterized in that, After generating a single file checksum for the data file, the method further includes: In the event that at least one fifth target data block is deleted from the data file, the first checksum of the fifth target data block and its corresponding first data range are deleted from the first verification layer of the data file, and the first data range corresponding to each first checksum in the first verification layer is redefined to update the first verification layer. Perform combined verification on each of the first check codes in the updated first verification layer, and update the second verification layer of the data file; Perform combined verification on each of the second check codes in the updated second verification layer, and update the third verification layer of the data file; The combined verification process continues until the file verification code of the data file is updated.

8. A data verification device, characterized in that, include: The system includes a data file acquisition unit, a first checksum generation unit, a second checksum generation unit, a third checksum generation unit, and a file checksum generation unit. The data file acquisition unit is used to acquire a data file; The first check code generation unit is used to split the data file into multiple data blocks, generate a first check code for each data block, and the first check code of each data block and its corresponding first data range constitute the first check layer of the data file. The second check code generation unit is used to perform combined verification on each of the first check codes in the first verification layer to generate multiple second check codes, and the second verification layer of the data file is composed of each second check code and its corresponding second data range. The third check code generation unit is used to perform combined verification on each of the second check codes in the second verification layer to generate multiple third check codes, and the third verification layer of the data file is composed of each of the third check codes and its corresponding third data range. The file verification code generation unit is used to continuously perform a combined verification process until a single file verification code for the data file is generated. The file verification code and its corresponding file data range constitute the topmost verification layer of the data file.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the data verification method as described in any one of claims 1 to 7.

10. An electronic device, the electronic device comprising at least one processor, and at least one memory and a bus connected to the processor; wherein, The processor and the memory communicate with each other via the bus; The processor is used to invoke program instructions in the memory to execute the data verification method as described in any one of claims 1 to 7.

Images