File verification method and device, storage medium and electronic equipment
By chunking files and generating multi-level identification using hashing algorithms to form verification files, the problem of low file verification efficiency is solved, and the rapid identification and update data blocks are realized and resource usage is reduced.
Patent Information
- Application Number
- CN202411045417.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
AI Technical Summary
The existing file verification methods are inefficient, especially when the files are large or large, which leads to an increase in time consumption and system resource usage, thereby reducing inspection efficiency.
By chunking the target file and generating multi-level identifiers using hash algorithms, the first and second-level verification files are formed, and the verification files are updated when the file is updated, the data block is determined through the comparison identifier, and the three-layer key value is used. Structural and omap data structures are used to improve inspection efficiency.
It effectively improves the efficiency of file verification, can quickly locate and identify file updated data blocks, reduces resource usage and time consumption, and solves the problem of low file verification efficiency.
Smart Images

Figure CN118916332A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of file verification technology, and specifically, to a file verification method, device, storage medium and electronic device. Background Art
[0002] With the improvement of digitalization, the requirements of various enterprises for data security are gradually increasing, and file verification is an indispensable and extremely important part of data security. Conventional verification methods usually require reading the entire file content of both parties for comparison. The verification time increases linearly with the size and number of files. When the files are large and the number of files is large, it will take a long time and occupy system resources, resulting in low file verification efficiency.
[0003] Therefore, in the related art, there is a problem of low file verification efficiency.
[0004] With regard to the problem of low efficiency of file verification in related technologies, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a file verification method, device, storage medium and electronic device to at least solve the problem of low file verification efficiency in the related art.
[0006] According to one embodiment of the present application, a file verification method is provided, including: dividing a target file into blocks according to a preset data size to obtain multiple data blocks, wherein the target file represents a file to be verified; calculating the multiple data blocks by a hash algorithm to obtain a first-level verification file; updating the first-level verification file after the target file is updated to obtain a second-level verification file; verifying the first-level verification file and the second-level verification file, and determining the updated data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0007] In an exemplary embodiment, the multiple data blocks are calculated by a hash algorithm to obtain a first-level verification file, including: calculating the multiple data blocks separately by a hash algorithm to obtain multiple third-level identifiers, wherein one third-level identifier corresponds to one data block; calculating a first preset number of third-level identifiers by the hash algorithm to obtain a second-level identifier; calculating a second preset number of second-level identifiers by the hash algorithm to obtain a first-level identifier; saving the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers according to a preset data structure to obtain the first-level verification file.
[0008] In an exemplary embodiment, the preset data structure is a three-layer key-value pair structure, and multiple first-level identifiers, multiple second-level identifiers, and the multiple third-level identifiers are saved according to the preset data structure to obtain the first-level verification file, including: determining each of the multiple third-level identifiers as a key of the third layer in the three-layer key-value pair structure, and determining the sequence number of the data block corresponding to each third-level identifier as the value of the third layer in the three-layer key-value pair structure to obtain multiple three-layer key-value pairs; determining each of the multiple second-level identifiers as a key of the second layer in the three-layer key-value pair structure, and determining the target three-layer key-value pair as the target The target layer 2 key-value pair is a value of the second layer in the three-layer key-value pair structure, wherein the key of the target layer 3 key-value pair belongs to the third-level identifier corresponding to each second-level identifier; each of the multiple first-level identifiers is determined as the key of the first layer in the three-layer key-value pair structure, and the target layer 2 key-value pair is determined as the value of the first layer in the three-layer key-value pair structure, wherein the key of the target layer 2 key-value pair belongs to the second-level identifier corresponding to each first-level identifier; after determining that the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers are all saved in the three-layer key-value pair structure, the three-layer key-value pair structure is determined as the first-level verification file.
[0009] In an exemplary embodiment, the first-level verification file is updated after the target file is updated to obtain a second-level verification file, including: receiving an update instruction for the target object, and updating the target data block according to the update instruction; calculating the updated data block by a hash algorithm to obtain an updated third-level identifier; determining the second-level identifier corresponding to the updated third-level identifier, and calculating the updated third-level identifier by a hash algorithm to obtain an updated second-level identifier; determining the first-level identifier corresponding to the updated second-level identifier, and calculating the updated second-level identifier by a hash algorithm to obtain an updated first-level identifier; replacing the third-level identifier before the update in the first-level verification file according to the updated third-level identifier, replacing the second-level identifier before the update in the first-level verification file according to the updated second-level identifier, and replacing the first-level identifier before the update in the first-level verification file according to the updated first-level identifier to obtain the second-level verification file.
[0010] In an exemplary embodiment, the first-level verification file and the second-level verification file are verified, and the update data block of the target file is determined according to different identifiers in the first-level verification file and the second-level verification file, including: comparing the first-level identifier in the first-level verification file with the first-level identifier in the second-level verification file, determining the first identifier from the first-level identifier in the first-level verification file, and determining the second identifier from the first-level identifier in the second-level verification file, wherein the comparison result of the first identifier and the second identifier is inconsistent; comparing the second-level identifier corresponding to the first identifier with the second-level identifier corresponding to the second identifier, determining the third identifier from the second-level identifier corresponding to the first identifier, and determining the fourth identifier from the second-level identifier corresponding to the second identifier, wherein the comparison result of the third identifier and the fourth identifier is inconsistent; comparing the third-level identifier corresponding to the third identifier with the third-level identifier corresponding to the fourth identifier, determining the fifth identifier from the third-level identifier corresponding to the third identifier, and determining the sixth identifier from the third-level identifier corresponding to the fourth identifier, wherein the comparison result of the fifth identifier and the sixth identifier is inconsistent; determining the update data block of the target file according to the data block corresponding to the fifth identifier and the data block corresponding to the sixth identifier.
[0011] In an exemplary embodiment, the method also includes: counting the update frequency of each third-level identifier in the second-level verification file; determining the target second-level identifier from the second-level verification file, wherein the update frequency of the third-level identifier corresponding to the target second-level identifier is less than the preset update frequency; and deleting the third-level identifier corresponding to the target second-level identifier from the second-level verification file.
[0012] In an exemplary embodiment, after deleting the third-level identifier corresponding to the target second-level identifier from the second-level verification file, the method also includes: when determining that the third-level identifier corresponding to the target second-level identifier is to be updated, recalculating the third-level identifier corresponding to the target second-level identifier to obtain the target third-level identifier; and adding the target third-level identifier to the second-level verification file.
[0013] According to another embodiment of the present application, a file verification device is provided, including: a block segmentation module, used to segment a target file according to a preset data size to obtain multiple data blocks, wherein the target file represents a file to be verified; a calculation module, used to calculate the multiple data blocks through a hash algorithm to obtain a first-level verification file; an update module, used to update the first-level verification file after the target file is updated to obtain a second-level verification file; a verification module, used to verify the first-level verification file and the second-level verification file, and determine the updated data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.
[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.
[0017] Through this application, the target file to be verified can be divided into multiple data blocks, and the multiple data blocks can be calculated by a hash algorithm to generate a first-level verification file. After the target file is updated, the first-level verification file is correspondingly updated to a second-level verification file. By comparing the first-level verification file and the second-level verification file, the updated data block of the target file can be determined. Therefore, the problem of low file verification efficiency in the related art can be solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a hardware structure block diagram of a server device of a file verification method in an embodiment of the present application;
[0019] Figure 2 is a flowchart of a file verification method according to an embodiment of the present application;
[0020] Figure 3 is a schematic diagram of a file verification method according to an embodiment of the present application (I);
[0021] Figure 4 is a schematic diagram of a file verification method according to an embodiment of the present application (II);
[0022] Figure 5 Schematic diagram of a file verification method according to an embodiment of the present application (III);
[0023] Figure 6 It is a structural block diagram of a file verification device according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0026] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a file verification method in an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the file verification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method for implementing the above-mentioned file verification. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0029] In this embodiment, a file verification method is provided. Figure 2 is a flowchart of a file verification method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0030] Step S202, dividing the target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified;
[0031] Optionally, in the above step S202, the preset data size is the size of the data block. For example, the target file can be divided into data blocks of 2kb, 4kb, and 8kb, which can be adjusted according to actual needs.
[0032] Step S204, calculating the multiple data blocks by a hash algorithm to obtain a first-level verification file;
[0033] Optionally, in the above step S202, the hash algorithm can adopt a CRC32 (Cyclic Redundancy Check 32-bit) algorithm. The CRC32 algorithm is a widely used algorithm for detecting errors in data transmission or storage. The CRC32 algorithm generates a 32-bit (i.e., 4 bytes) check code by calculating a polynomial of a data block.
[0034] Step S206, updating the first-level verification file after the target file is updated to obtain a second-level verification file;
[0035] Step S208: verify the first-level verification file and the second-level verification file, and determine the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0036] Through the above steps, the target file to be verified can be divided into multiple data blocks, and the first-level verification file is generated by calculating the multiple data blocks through the hash algorithm. After the target file is updated, the first-level verification file is correspondingly updated to the second-level verification file, and the updated data block of the target file can be determined by comparing the first-level verification file and the second-level verification file. Therefore, the problem of low file verification efficiency in the related art can be solved.
[0037] In an exemplary embodiment, the specific implementation process of the above-mentioned step S204 calculating the multiple data blocks through a hash algorithm to obtain a first-level verification file includes: calculating the multiple data blocks respectively through a hash algorithm to obtain multiple third-level identifiers, wherein one third-level identifier corresponds to one data block; calculating a first preset number of third-level identifiers through the hash algorithm to obtain a second-level identifier; calculating a second preset number of second-level identifiers through the hash algorithm to obtain a first-level identifier; saving the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers according to a preset data structure to obtain the first-level verification file.
[0038] Optionally, in the above embodiment, for example, the target file is divided into multiple data blocks of 4kb in size, and the CRC32 algorithm can be used to calculate each data to generate a 4-byte third-level identifier, that is, a total of (file size / 4kb) third-level identifiers are generated. For example, a file of 1G size will produce (1024*1024kb) / 4kb=262144 third-level identifiers, which is still a large value. Therefore, the CRC32 algorithm can be used to calculate multiple third-level identifiers to generate a second-level identifier, and one second-level identifier can correspond to a maximum of 10240 third-level identifiers. The CRC32 algorithm can be used to calculate multiple second-level identifiers to generate a first-level identifier, and one first-level identifier can correspond to a maximum of 10240 second-level identifiers.
[0039] In an exemplary embodiment, the preset data structure is a three-layer key-value pair structure. The above-mentioned multiple first-level identifiers, multiple second-level identifiers, and the multiple third-level identifiers are saved according to the preset data structure to obtain the implementation process of the first-level verification file, which specifically includes: determining each of the multiple three-level identifiers as a key of the third layer in the three-layer key-value pair structure, and determining the sequence number of the data block corresponding to each three-level identifier as the value of the third layer in the three-layer key-value pair structure to obtain multiple three-layer key-value pairs; determining each of the multiple second-level identifiers as a key of the second layer in the three-layer key-value pair structure, and determining the target three-layer key value. Determine the value of the second layer in the three-layer key-value pair structure, wherein the key of the target three-layer key-value pair belongs to the third-level identifier corresponding to each of the second-level identifiers; determine each of the multiple first-level identifiers as the key of the first layer in the three-layer key-value pair structure, and determine the target second-layer key-value pair as the value of the first layer in the three-layer key-value pair structure, wherein the key of the target second-layer key-value pair belongs to the second-level identifier corresponding to each of the first-level identifiers; after determining that the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers are all saved in the three-layer key-value pair structure, determine the three-layer key-value pair structure as the first-level verification file.
[0040] Optionally, in the above embodiment, the above three-layer key-value pair structure can use an omap (Object Map) data structure, which is used to store key-value pairs. It has a multi-layer structure, each layer is stored in a map, and the hierarchical structure of omap is similar to a tree, and each layer of the map stores the identifier of the next layer. This organization method helps to quickly access and update the identifier of a specific data block.
[0041] Through the above embodiment, the omap data structure is used as a persistent object of the hierarchical verification file, and the hierarchical tree structure is utilized to effectively improve the file verification efficiency.
[0042] In an exemplary embodiment, the above-mentioned step S206 updates the first-level verification file after the target file is updated to obtain the second-level verification file, which can be implemented in the following manner: receiving an update instruction for the target object, and updating the target data block according to the update instruction; calculating the updated data block through a hash algorithm to obtain an updated third-level identifier; determining the second-level identifier corresponding to the updated third-level identifier, and calculating the updated third-level identifier through a hash algorithm to obtain an updated second-level identifier; determining the first-level identifier corresponding to the updated second-level identifier, and calculating the updated second-level identifier through a hash algorithm to obtain an updated first-level identifier; replacing the third-level identifier before the update in the first-level verification file according to the updated third-level identifier, replacing the second-level identifier before the update in the first-level verification file according to the updated second-level identifier, and replacing the first-level identifier before the update in the first-level verification file according to the updated first-level identifier to obtain the second-level verification file.
[0043] Optionally, in the above embodiment, Figure 3 For example, Figure 3 The file in (equivalent to the target file) is divided into multiple blocks (equivalent to data blocks). When the business process receives the update instruction of the target object, it starts to modify the file. The minimum modification unit is block. In this embodiment, the size of a block is 4kb. Figure 3 As shown in the figure, for example, if a byte is modified in the 8th block, the business process first reads the 4kb data in the block into the cache, then overwrites the one-byte data at the location that needs to be modified to build a new block, and then writes the new block to the file. When an additional write occurs, the business process will build a new block in the cache with the data that needs to be written. Figure 3 An additional write of 4 blocks occurs at the end of the file. The 13th, 14th, 15th, and 16th blocks are first constructed in the cache, and then the constructed blocks are written to the file.
[0044] exist Figure 3In the process, when a file is updated, the business process notifies the identity calculation process that a block has been modified or added. The identity calculation process uses shared memory to read the modified or added block in the business process cache, and then uses the crc32 hash algorithm to calculate the identity of the modified or added block (equivalent to the third-level identification). Each block of the file corresponds to an identity. When the identity calculation process reads the newly added block, it pushes the identification calculation task to the task queue for asynchronous execution. When the business process is busy, the execution speed of the task queue will be limited, so as not to preempt the resources of the business process. The identity of the newly added block will be recorded in the hierarchical verification file of the omap structure. Each file corresponds to a hierarchical verification file. When performing data consistency verification, you only need to compare the hierarchical verification file.
[0045] Optionally, in the above embodiment, Figure 4 For example, the smallest unit for file update is block. Whenever a block changes, its identity will be recalculated by the identity calculation process, that is, a 4-byte third-level identity will be generated after the crc32 check. After the third-level identity is recalculated, the changed content will be passed to the second-level identity, thereby triggering the update of the second-level identity. Each second-level identity is calculated by the crc32 hash value of its corresponding third-level identity. For example, the third-level identity corresponding to the first block is converted into an integer type of 12345, and the third-level identity corresponding to the second block is converted into an integer type of 67890. The identity calculation process concatenates the third-level identity into "12345" + "67890" + "...." = "1234567890..." as the input of crc32 to generate a 4-byte second-level identity. Whenever a third-level identity is updated, the second-level identity is updated accordingly; when a second-level identity is updated, the first-level identity is updated according to the same rule.
[0046] In an exemplary embodiment, the above step S208 verifies the first-level verification file and the second-level verification file, and determines the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file, specifically including: comparing the first-level identifier in the first-level verification file with the first-level identifier in the second-level verification file, determining the first identifier from the first-level identifier in the first-level verification file, and determining the second identifier from the first-level identifier in the second-level verification file, wherein the comparison result of the first identifier and the second identifier is inconsistent; comparing the second-level identifier corresponding to the first identifier with the second-level identifier corresponding to the second identifier, determining the third identifier from the second-level identifier corresponding to the first identifier, and determining the fourth identifier from the second-level identifier corresponding to the second identifier, wherein the comparison result of the third identifier and the fourth identifier is inconsistent; comparing the third-level identifier corresponding to the third identifier with the third-level identifier corresponding to the fourth identifier, determining the fifth identifier from the third-level identifier corresponding to the third identifier, and determining the sixth identifier from the third-level identifier corresponding to the fourth identifier, wherein the comparison result of the fifth identifier and the sixth identifier is inconsistent; determining the update data block of the target file according to the data block corresponding to the fifth identifier and the data block corresponding to the sixth identifier.
[0047] Optionally, in the above embodiment, the identity calculation process pushes the level identification calculation to the task queue for asynchronous execution every time a change occurs in the file unit. When all changes to the file are calculated, the timestamp of the last modification of the file is used as the verification completion mark and saved in the attributes of the level verification file. When a consistency check is required, first determine whether the best modification time of the file and the verification completion mark are consistent. If they are consistent, it means that the file has updated the level verification file and the level verification file is available; if they are inconsistent, it means that the file has not updated the level verification file, and the identity calculation process is notified to perform a level identification check update on the file.
[0048] Optionally, in the above embodiment, for example, the current version and the historical version of the file are verified. When the hierarchical verification files of both parties are available, a comparison is performed from the first-level identifier. If the first-level identifiers are the same, it means that the second-level and third-level identifiers under the identifier are the same, and there is no need to verify downward. If all the first-level identifiers are the same, it means that the file contents of both parties are consistent; if a certain first-level identifier is different, then continue to verify downward to find a different second-level identifier, and then continue to verify downward to find a different third-level identifier, so that the different contents of the files of the two parties can be clearly located, and then feedback is given to the business caller.
[0049] Through the above-mentioned embodiment, asynchronous hash calculation is performed on the updated file without occupying business resources. Through multi-level identification, different contents can be accurately located during file verification, and verification efficiency can be improved.
[0050] In an exemplary embodiment, the method also includes: counting the update frequency of each third-level identifier in the second-level verification file; determining the target second-level identifier from the second-level verification file, wherein the update frequency of the third-level identifier corresponding to the target second-level identifier is less than the preset update frequency; and deleting the third-level identifier corresponding to the target second-level identifier from the second-level verification file.
[0051] In an exemplary embodiment, after deleting the third-level identifier corresponding to the target second-level identifier from the second-level verification file, the method further includes: when determining that the third-level identifier corresponding to the target second-level identifier is to be updated, recalculating the third-level identifier corresponding to the target second-level identifier to obtain the target third-level identifier; and adding the target third-level identifier to the second-level verification file.
[0052] Optionally, in the above embodiment, each file corresponds to a hierarchical verification file, most of the content of the hierarchical verification file is on the third-level identifier, each block (taking 4kb as an example) corresponds to a 4-byte verification file, and the content ratio of the file and the verification file is 1024:1. For example, the verification file size corresponding to a 1G file is 1M in size. When about 1T of data is stored on the disk, the verification file will occupy approximately 1G of space. However, in actual business scenarios, files are rarely modified on a large scale, and most third-level identifiers rarely change, and do not require persistent storage. In order to improve disk utilization, the present application designs a mechanism for streamlining based on access popularity for the third-level identifiers, which can greatly reduce the size of the hierarchical verification file.
[0053] like Figure 5 As shown, the identity calculation process will periodically calculate the heat value when updating the secondary identification. The heat value is determined by the update frequency of the third-level identification corresponding to the secondary identification. This embodiment adopts an algorithm that dynamically decays the heat over time to calculate the heat. When the block corresponding to the third-level identification is accessed, the heat value is increased by 1; if it is not accessed for more than 1 minute, it decays according to the decay algorithm, for example, the heat value decreases by 0.5 per minute. When the heat value decreases to the preset value, all third-level identifications under the secondary identification are deleted. If the block corresponding to the deleted third-level identification is modified again, the third-level identification needs to be repaired, and all the third-level identifications under the second-level identification to which the block belongs need to be recalculated, and then the second-level identification is updated based on the recalculated third-level identification, and then the first-level identification is updated.
[0054] The above embodiment deletes the infrequently accessed third-level identifications by dynamically calculating the heat value. Although more resources are required to calculate the deleted third-level identifications when the third-level identifications are repaired, the probability of identification repair occurring under normal business is very low. Therefore, the benefits far outweigh the losses, greatly improving the utilization rate of the disk space.
[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0056] In the present embodiment, a file verification device is also provided, and the system is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions that have been made are not repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0057] Figure 6 is a structural block diagram of a file verification device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:
[0058] A block division module 62, used to divide the target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified;
[0059] A calculation module 64, configured to calculate the plurality of data blocks by using a hash algorithm to obtain a first-level verification file;
[0060] An updating module 66, configured to update the first-level verification file after the target file is updated to obtain a second-level verification file;
[0061] The verification module 68 is used to verify the first-level verification file and the second-level verification file, and determine the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0062] Through the above device, the target file to be verified can be divided into multiple data blocks, and the multiple data blocks can be calculated by the hash algorithm to generate a first-level verification file. After the target file is updated, the first-level verification file is correspondingly updated to the second-level verification file. By comparing the first-level verification file and the second-level verification file, the updated data block of the target file can be determined. Therefore, the problem of low file verification efficiency in the related art can be solved.
[0063] In an exemplary embodiment, the calculation module 64 is also used to: calculate the multiple data blocks respectively through a hash algorithm to obtain multiple third-level identifiers, wherein one third-level identifier corresponds to one data block; calculate a first preset number of third-level identifiers through the hash algorithm to obtain a second-level identifier; calculate a second preset number of second-level identifiers through the hash algorithm to obtain a first-level identifier; save the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers according to a preset data structure to obtain the first-level verification file.
[0064] In an exemplary embodiment, the calculation module 64 is further used to: determine each of the multiple three-level identifiers as a key of the third layer in the three-layer key-value pair structure, determine the sequence number of the data block corresponding to each of the three-level identifiers as the value of the third layer in the three-layer key-value pair structure, and obtain multiple three-layer key-value pairs; determine each of the multiple second-level identifiers as a key of the second layer in the three-layer key-value pair structure, and determine the target three-layer key-value pair as the value of the second layer in the three-layer key-value pair structure, wherein the key of the target three-layer key-value pair belongs to the third-level identifier corresponding to each of the second-level identifiers; determine each of the multiple first-level identifiers as a key of the first layer in the three-layer key-value pair structure, and determine the target second-layer key-value pair as the value of the first layer in the three-layer key-value pair structure, wherein the key of the target second-layer key-value pair belongs to the second-level identifier corresponding to each of the first-level identifiers; after determining that the multiple first-level identifiers, the multiple second-level identifiers, and the multiple third-level identifiers are all saved in the three-layer key-value pair structure, determine the three-layer key-value pair structure as the first-level verification file.
[0065] In an exemplary embodiment, the update module 66 is also used to: receive an update instruction for a target object, and update a target data block according to the update instruction; calculate the updated data block through a hash algorithm to obtain an updated third-level identifier; determine the second-level identifier corresponding to the updated third-level identifier, and calculate the updated third-level identifier through a hash algorithm to obtain an updated second-level identifier; determine the first-level identifier corresponding to the updated second-level identifier, and calculate the updated second-level identifier through a hash algorithm to obtain an updated first-level identifier; replace the third-level identifier before update in the first-level verification file according to the updated third-level identifier, replace the second-level identifier before update in the first-level verification file according to the updated second-level identifier, and replace the first-level identifier before update in the first-level verification file according to the updated first-level identifier to obtain the second-level verification file.
[0066] In an exemplary embodiment, the verification module 68 is also used to: compare the first-level identifier in the first-level verification file with the first-level identifier in the second-level verification file, determine the first identifier from the first-level identifier in the first-level verification file, and determine the second identifier from the first-level identifier in the second-level verification file, wherein the comparison results of the first identifier and the second identifier are inconsistent; compare the second-level identifier corresponding to the first identifier with the second-level identifier corresponding to the second identifier, determine the third identifier from the second-level identifier corresponding to the first identifier, and determine the fourth identifier from the second-level identifier corresponding to the second identifier, wherein the comparison results of the third identifier and the fourth identifier are inconsistent; compare the third-level identifier corresponding to the third identifier with the third-level identifier corresponding to the fourth identifier, determine the fifth identifier from the third-level identifier corresponding to the third identifier, and determine the sixth identifier from the third-level identifier corresponding to the fourth identifier, wherein the comparison results of the fifth identifier and the sixth identifier are inconsistent; determine the update data block of the target file according to the data block corresponding to the fifth identifier and the data block corresponding to the sixth identifier.
[0067] In an exemplary embodiment, the device is also used to: count the update frequency of each third-level identifier in the second-level verification file; determine the target second-level identifier from the second-level verification file, wherein the update frequency of the third-level identifier corresponding to the target second-level identifier is less than the preset update frequency; and delete the third-level identifier corresponding to the target second-level identifier from the second-level verification file.
[0068] In an exemplary embodiment, the device is also used to: when it is determined that the third-level identifier corresponding to the target secondary identifier is updated, recalculate the third-level identifier corresponding to the target secondary identifier to obtain the target third-level identifier; and add the target third-level identifier to the second-level verification file.
[0069] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0070] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.
[0071] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:
[0072] S1, dividing a target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified;
[0073] S2, calculating the multiple data blocks by a hash algorithm to obtain a first-level verification file;
[0074] S3, updating the first-level verification file after the target file is updated to obtain a second-level verification file;
[0075] S4, verifying the first-level verification file and the second-level verification file, and determining the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0076] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0077] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0078] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0079] S1, dividing a target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified;
[0080] S2, calculating the multiple data blocks by a hash algorithm to obtain a first-level verification file;
[0081] S3, updating the first-level verification file after the target file is updated to obtain a second-level verification file;
[0082] S4, verifying the first-level verification file and the second-level verification file, and determining the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0083] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.
[0084] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:
[0085] S1, dividing a target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified;
[0086] S2, calculating the multiple data blocks by a hash algorithm to obtain a first-level verification file;
[0087] S3, updating the first-level verification file after the target file is updated to obtain a second-level verification file;
[0088] S4, verifying the first-level verification file and the second-level verification file, and determining the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
[0089] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0090] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0091] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.
Claims
1. A file verification method, characterized in that: include: Dividing the target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified; Calculate the multiple data blocks by using a hash algorithm to obtain a first-level verification file; After the target file is updated, the first-level verification file is updated to obtain a second-level verification file; The first-level verification file and the second-level verification file are verified, and the update data block of the target file is determined according to different identifiers in the first-level verification file and the second-level verification file.
2. The method according to claim 1, characterized in that The plurality of data blocks are calculated by a hash algorithm to obtain a first-level verification file, including: Calculating the multiple data blocks respectively by a hash algorithm to obtain multiple third-level identifiers, wherein one third-level identifier corresponds to one data block; Calculate a first preset number of third-level identifiers by using the hash algorithm to obtain a second-level identifier; Calculate a second preset number of secondary identifiers by using the hash algorithm to obtain a primary identifier; The plurality of first-level identifiers, the plurality of second-level identifiers, and the plurality of third-level identifiers are saved according to a preset data structure to obtain the first-level verification file.
3. The method according to claim 2, characterized in that The preset data structure is a three-layer key-value pair structure, and multiple first-level identifiers, multiple second-level identifiers, and multiple third-level identifiers are saved according to the preset data structure to obtain the first-level verification file, including: Determine each of the multiple three-level identifiers as a key of the third layer in the three-layer key-value pair structure, and determine the sequence number of the data block corresponding to each of the three-level identifiers as a value of the third layer in the three-layer key-value pair structure, to obtain multiple three-layer key-value pairs; Determine each of the multiple secondary identifiers as a key of the second layer in the three-layer key-value pair structure, and determine a target three-layer key-value pair as a value of the second layer in the three-layer key-value pair structure, wherein the key of the target three-layer key-value pair belongs to the third-level identifier corresponding to each of the secondary identifiers; Determine each of the multiple first-level identifiers as a key of the first layer in the three-layer key-value pair structure, and determine a target second-layer key-value pair as a value of the first layer in the three-layer key-value pair structure, wherein the key of the target second-layer key-value pair belongs to the second-level identifier corresponding to each of the first-level identifiers; After determining that the plurality of first-level identifiers, the plurality of second-level identifiers, and the plurality of third-level identifiers are all saved in the three-layer key-value pair structure, the three-layer key-value pair structure is determined as the first-level verification file.
4. The method according to claim 3, characterized in that After the target file is updated, the first-level verification file is updated to obtain a second-level verification file, including: receiving an update instruction of a target object, and updating a target data block according to the update instruction; The updated data block is calculated by a hash algorithm to obtain an updated third-level identifier; Determine the secondary identification corresponding to the updated tertiary identification, and calculate the updated tertiary identification by a hash algorithm to obtain an updated secondary identification; Determine the primary identifier corresponding to the updated secondary identifier, and calculate the updated secondary identifier by a hash algorithm to obtain an updated primary identifier; Replace the third-level identifier before updating in the first-level verification file according to the updated third-level identifier, replace the second-level identifier before updating in the first-level verification file according to the updated second-level identifier, and replace the first-level identifier before updating in the first-level verification file according to the updated first-level identifier to obtain the second-level verification file.
5. The method according to claim 4, characterized in that Verifying the first-level verification file and the second-level verification file, and determining the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file, including: comparing the first-level identifier in the first-level verification file with the first-level identifier in the second-level verification file, determining a first identifier from the first-level identifier in the first-level verification file, and determining a second identifier from the first-level identifier in the second-level verification file, wherein the comparison result between the first identifier and the second identifier is inconsistent; Comparing the secondary identifier corresponding to the first identifier with the secondary identifier corresponding to the second identifier, determining a third identifier from the secondary identifier corresponding to the first identifier, and determining a fourth identifier from the secondary identifier corresponding to the second identifier, wherein the comparison result between the third identifier and the fourth identifier is inconsistent; Comparing the third-level identification corresponding to the third identification with the third-level identification corresponding to the fourth identification, determining a fifth identification from the third-level identification corresponding to the third identification, and determining a sixth identification from the third-level identification corresponding to the fourth identification, wherein the comparison result between the fifth identification and the sixth identification is inconsistent; The updated data block of the target file is determined according to the data block corresponding to the fifth identifier and the data block corresponding to the sixth identifier.
6. The method according to claim 4, characterized in that The method further comprises: Counting the update frequency of each third-level identifier in the second-level verification file; Determine a target secondary identifier from the second-level verification file, wherein an update frequency of a third-level identifier corresponding to the target secondary identifier is less than a preset update frequency; The third-level identifier corresponding to the target second-level identifier is deleted from the second-level verification file.
7. The method according to claim 6, characterized in that After deleting the third-level identifier corresponding to the target second-level identifier from the second-level verification file, the method further includes: In the case where it is determined that the third-level identifier corresponding to the target second-level identifier is to be updated, the third-level identifier corresponding to the target second-level identifier is recalculated to obtain the target third-level identifier; The target third-level identifier is added to the second-level verification file.
8. A file verification device, characterized in that: include: A block division module, used for dividing a target file into blocks according to a preset data size to obtain a plurality of data blocks, wherein the target file represents a file to be verified; A calculation module, used for calculating the multiple data blocks by using a hash algorithm to obtain a first-level verification file; An updating module, configured to update the first-level verification file after the target file is updated to obtain a second-level verification file; A verification module is used to verify the first-level verification file and the second-level verification file, and determine the update data block of the target file according to different identifiers in the first-level verification file and the second-level verification file.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Verifying method for network file transmission
CN102325167A
Nested key-value storage method and device
CN107741833A
Block chain state data caching and querying method and device, and storage medium
CN110175188A
Method, apparatus and computer program product for data management
CN113590020A
Electronic archive file rapid verification method, system, device and medium
CN116150092A
Cited By
Method, device and equipment for checking data between network hierarchies, medium and product
CN119766391A
A network inter-level data checking method, device, equipment, medium and product
CN119766391B
