Network disk file synchronization method based on dynamic hash detection

Abstract

The invention provides a network disk file synchronization method based on dynamic hash detection, and relates to the technical field of computers. The method comprises the following steps: a client obtains a to-be-synchronized file, and determines a corresponding Hash calculation strategy according to the size of the to-be-synchronized file; the client calculates a Hash feature value of the to-be-synchronized file based on the determined Hash calculation strategy; the client sends the Hash feature value to a cloud server, wherein the cloud server stores a Hash fingerprint database; the cloud server queries the Hash fingerprint database and judges whether a target record matched with the Hash feature value of the to-be-synchronized file exists or not; and completing file synchronization according to whether the target record exists in the Hash fingerprint database. Compared with the prior art, the method has the advantages that efficient duplicate file detection in network disk file synchronization is realized through a hierarchical hash calculation strategy, a cloud hash fingerprint database dynamic updating mechanism and multi-dimensional conflict verification, redundant data transmission is reduced, and the network disk file synchronization efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a network disk file synchronization method based on dynamic hash detection. BACKGROUND

[0002] In the file synchronization scenario of a traditional network disk, repeated file detection generally adopts a full hash calculation scheme (such as MD5, SHA-1 algorithm, etc.). The core logic is: the hash value of the complete content of all files to be synchronized is calculated, and the results are compared with the server-side data one by one; at the same time, this scheme does not identify and process the differences in the internal data blocks of the file, resulting in the following obvious defects and deficiencies in actual application:

[0003] 1. Low efficiency and significant time consumption of large file hash calculation: since the hash operation needs to be performed on the complete content of the file, the larger the file size, the longer the calculation time. Taking a common 1GB file as an example, a single full hash calculation usually takes 3-5 seconds, and if a large number of files are synchronized, it will seriously slow down the overall synchronization progress and affect the user experience.

[0004] 2. Redundant data transmission and waste of bandwidth resources: this scheme does not identify the repetition of file data blocks, even if different files contain the same data segments (such as two video clips sharing the same clip segment), these repeated segments will still be uploaded with the file multiple times, causing a large amount of bandwidth resources to be wasted, and increasing the receiving and storage pressure of the server.

[0005] 3. High risk of misjudgment due to reliance on single hash value comparison: only by matching the single hash value can the file be determined to be repeated or not, which has the problem of misjudgment due to "hash collision", that is, two completely different files may generate the same hash value due to algorithm characteristics, and then be incorrectly determined as repeated files; on the contrary, it may also miss the identification of actual repeated files due to small differences in hash values, affecting the synchronization accuracy. SUMMARY

[0006] The purpose of the present application is to provide a network disk file synchronization method based on dynamic hash detection, which can improve the efficiency and accuracy of network disk synchronization.

[0007] The present application provides a network disk file synchronization method based on dynamic hash detection, comprising:

[0008] The client obtains a file to be synchronized, and determines a corresponding hash calculation strategy according to the size of the file to be synchronized;

[0009] The client calculates the hash feature value of the file to be synchronized based on the determined hash calculation strategy;

[0010] The client sends the hash feature value to a cloud server, and the cloud server stores a hash fingerprint library, each record of which contains a file unique identifier, a file feature, a file physical storage address, and a reference count; wherein the file feature is a hash feature value calculated according to the size of the corresponding file by using a corresponding hash calculation strategy;

[0011] The cloud server queries the hash fingerprint library to determine whether there is a target record matching the hash feature value of the file to be synchronized;

[0012] If the target record exists, the cloud server obtains the metadata of the file to be synchronized and compares it with the file metadata corresponding to the target record, the metadata at least including one of the file name, the file size, the file type, the file last modification time, and the file creation time; if the metadata is completely consistent, the client establishes an association with the file physical storage address in the target record, the cloud server adds 1 to the reference count of the target record, and the synchronization of the file to be synchronized is completed; if the metadata is not completely consistent, it is determined as hash collision, the client uploads the file to be synchronized to the cloud server, the cloud server stores the file to be synchronized to a new physical storage address, and adds a new record in the hash fingerprint library, the file feature of the new record being the hash feature value of the file to be synchronized, the file physical storage address being the new physical storage address, and the reference count being 1, and the synchronization of the file to be synchronized is completed.

[0013] If the target record does not exist, the client uploads the file to be synchronized to the cloud server, the cloud server stores the file to be synchronized to a new physical storage address, and adds a new record in the hash fingerprint library, the file feature of the new record being the hash feature value of the file to be synchronized, the file physical storage address being the new physical storage address, and the reference count being 1, and the synchronization of the file to be synchronized is completed.

[0014] In a possible implementation manner, the corresponding hash calculation strategy is determined according to the size of the file to be synchronized, including:

[0015] If the size of the file to be synchronized is less than 1MB, it is determined to use a full hash calculation strategy;

[0016] If the size of the file to be synchronized is greater than or equal to 1MB and less than or equal to 1GB, it is determined to use a block sampling hash calculation strategy;

[0017] If the size of the file to be synchronized is greater than 1GB, it is determined to use a calculation strategy combining metadata and incremental block hash.

[0018] In a possible implementation, the full-content hash calculation strategy is that the client calculates a hash value of the full content of the file to be synchronized by using a preset hash function, and the hash value is the hash feature value of the file to be synchronized.

[0019] In a possible implementation, the block sampling hash calculation strategy is that:

[0020] The client logically divides the file to be synchronized into data blocks of a fixed size, to obtain a total number of data blocks of the file to be synchronized.

[0021] The first preset proportion a of the data blocks in the file to be synchronized is extracted, the second preset proportion b of the data blocks in the middle of the file to be synchronized is extracted, and the third preset proportion c of the data blocks at the end of the file to be synchronized is extracted, where the number of the first preset proportion a of the data blocks is the upward rounding of the product of a and the total number of data blocks, the start offset of the second preset proportion b of the data blocks in the middle of the file to be synchronized is equal to , the number of the second preset proportion b of the data blocks is the upward rounding of the product of b and the total number of data blocks, and the start offset of the third preset proportion c of the data blocks at the end of the file to be synchronized is equal to , the number of the third preset proportion c of the data blocks is the upward rounding of the product of c and the total number of data blocks; and Size is the size of the file to be synchronized.

[0022] The hash values of the extracted data blocks are calculated by using a preset hash function, the hash values of all the extracted data blocks are spliced in sequence, the hash value of the spliced result is calculated by using the preset hash function again, to generate a sampling hash value, and the sampling hash value is the hash feature value of the file to be synchronized.

[0023] In a possible implementation, the combined metadata and incremental block hash calculation strategy is that:

[0024] The client calculates a hash value of the metadata of the file to be synchronized by using a preset hash function, to obtain a metadata hash value H_meta of the file to be synchronized.

[0025] The file to be synchronized is divided into data blocks of a fixed size, the hash values of the data blocks are calculated by using a preset hash function, the hash values of all the data blocks are combined, to form an incremental block hash list H_incremental, and the hash feature value of the file to be synchronized is (H_meta, H_incremental).

[0026] In a possible implementation, the method further includes:

[0027] When it is determined that there is a hash collision, after the client uploads the file to be synchronized to the cloud server, the cloud server recalculates the hash feature value of the uploaded file to be synchronized, and performs consistency verification with the hash feature value sent by the client, and after the verification passes, the operation of adding a record in the hash fingerprint library is performed.

[0028] In a possible implementation manner, the method further includes:

[0029] When the user performs a file deletion operation, the client sends a deletion request to the cloud server, the cloud server finds a target record corresponding to the file, reduces a reference count of the target record by 1, if the reference count is reduced to 0, the target record and the file in the file physical storage address corresponding to the target record are deleted, if the reference count is not reduced to 0, only the association between the user and the file physical storage address is deleted, and the target record and the file in the file physical storage address corresponding to the target record are retained.

[0030] In a possible implementation manner, the method further includes:

[0031] The cloud server periodically optimizes the hash fingerprint library, and deletes invalid records, the invalid records being records with a reference count of 0 or a file in a file physical storage address corresponding to the record being deleted.

[0032] In a possible implementation manner, the method further includes:

[0033] Data transmission between the client and the cloud server adopts an encryption protocol, the encryption protocol being an SSL / TLS protocol, to ensure security of the hash feature value, the metadata and the file content of the file to be synchronized in a transmission process.

[0034] Compared with the prior art, the cloud disk file synchronization method based on dynamic hash detection provided in the application dynamically adjusts a hash calculation strategy according to a file size; a cloud end establishes a hash fingerprint library to store historical file hash features, and a new file is queried against the hash fingerprint library before being uploaded, and if a match is found, a cloud file address is directly referenced; file metadata and hash value double-checking is introduced to avoid hash collision misjudgment. Compared with the prior art, the application realizes efficient repeated file detection in cloud disk file synchronization through a hierarchical hash calculation strategy, a cloud end hash fingerprint library dynamic updating mechanism and multi-dimensional conflict checking, reduces redundant data transmission, and improves cloud disk file synchronization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not considered a limitation of the application. Furthermore, like reference numerals are used to denote identical components across the drawings. In the drawings:

[0036] Figure 1 A flowchart of a cloud disk file synchronization method based on dynamic hash detection is shown;

[0037] Figure 2A synchronization flowchart of the file when the target record exists in the hash fingerprint library in step S105 is shown;

[0038] Figure 3 A synchronization flowchart of the file when the target record does not exist in the hash fingerprint library in step S105 is shown. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and so that the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0040] It should be noted that, unless otherwise specified, technical terms or scientific terms used in the present application should be understood as their common meanings to those skilled in the art to which the present application pertains.

[0041] In addition, the terms "first" and "second" and the like are used to distinguish different objects, rather than to describe a particular order. Furthermore, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a list of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed or can optionally further include other steps or units inherent to such processes, methods, products or devices.

[0042] Reference will be made to Figure 1 , Figure 1 is a flowchart of a network disk file synchronization method based on dynamic hash detection provided by an embodiment of the present application, comprising the following steps S101-S106:

[0043] S101, the client acquires a file to be synchronized, and determines a corresponding hash calculation strategy according to the size of the file to be synchronized;

[0044] Specifically, the step S101 of determining the corresponding hash calculation strategy according to the size of the file to be synchronized comprises:

[0045] If the size of the file to be synchronized is less than 1MB (small file), it is determined to use a full hash calculation strategy;

[0046] If the size of the file to be synchronized is greater than or equal to 1MB and less than or equal to 1GB (medium file), it is determined to use a block sampling hash calculation strategy;

[0047] If the size of the file to be synchronized is greater than 1GB (large file), it is determined to use a calculation strategy combining metadata and incremental block hash.

[0048] S102、client based on the determined hash calculation strategy to calculate the hash characteristic value of the file to be synchronized;

[0049] 1, the full amount of hash calculation strategy is:

[0050] The client uses a preset hash function (such as SHA-1) to perform hash calculation on the complete content of the file to be synchronized, and generates a unique full amount of hash value H_full, which is the hash characteristic value of the file to be synchronized.

[0051] For example: a 512KB Word document (.docx), read the entire file content, calculate its SHA-1 value H_full, which is used for subsequent repetitive comparison.

[0052] 2, the block sampling hash calculation strategy is:

[0053] The client logically divides the file to be synchronized into fixed-size data blocks, and obtains the total number of blocks of the file to be synchronized.

[0054] Extract the first preset proportion of data blocks in the file to be synchronized, the second preset proportion of data blocks in the middle and the third preset proportion of data blocks at the back, wherein the number of the first preset proportion a data blocks is the upward rounding of the product of a and the total number of blocks, the starting offset of the second preset proportion b data blocks in the middle is equal to , the number of b and the total number of blocks is the upward rounding, and the starting offset of the third preset proportion c data blocks at the back is equal to , the number of c and the total number of blocks is the upward rounding, and the size is the size of the file to be synchronized.

[0055] The hash values of the extracted data blocks are calculated by using a preset hash function, and the hash values of all the extracted data blocks are spliced in order, and then the hash value of the spliced result is calculated by using the preset hash function again to generate a sampling hash value, which is the hash characteristic value of the file to be synchronized.

[0056] Specifically, the file is logically divided into fixed-size data blocks (for example, the block size BlockSize = 4KB). Then, according to the preset sampling ratio, only the hash values of the data blocks at the following specific positions are calculated:

[0057] The first 10% of the blocks: calculate the continuous blocks at the starting position of the file .

[0058] The middle 5% of the blocks: calculate the continuous blocks near the middle position of the file , .

[0059] Last 10% of blocks: Calculate the hash of the last 12800 blocks (consecutive). , .

[0060] Concatenate or hash again all the hashes of the sampled blocks in order.

[0061] For example:

[0062] A 500MB PDF file.

[0063] Assume BlockSize = 4KB, total blocks ≈ 128000 blocks.

[0064] First 10%: Calculate the hash of the first 12800 blocks (consecutive).

[0065] Middle 5%: Calculate the hash of the 6400 blocks starting from the 60800th block (consecutive).

[0066] Last 10%: Calculate the hash of the 12800 blocks starting from the 115200th block (128000 - 12800 = 115200) (consecutive).

[0067] Concatenate the hashes of these 12800 + 6400 + 12800 = 32000 blocks in order, and calculate the SHA-1 again to get the final H_sample, which represents the characteristics of the key part of the file.

[0068] 3. The calculation strategy of combining metadata and incremental block hash is:

[0069] The client uses a preset hash function to hash the metadata of the file to be synchronized, obtaining the metadata hash value H_meta of the file to be synchronized.

[0070] Divide the file to be synchronized into data blocks of a fixed size, and use a preset hash function to calculate the hash value of each data block. Combine the hash values of all data blocks to form an incremental block hash list H_incremental. The hash feature value of the file to be synchronized is (H_meta, H_incremental).

[0071] The specific process is as follows:

[0072] Metadata Hash (H meta): Compute the hash of the file's key metadata. This usually includes: file name (or path), file size (Size), last modified time (Mtime), file type (extension), etc. For example: H meta = SHA-1(file name + str(Size) + str(Mtime) + file extension). This hash value changes very quickly, as soon as one item of metadata is changed (such as renaming, modifying timestamp), H meta is different.

[0073] Incremental Chunk Hash (H incremental): The system maintains a history state for each file (if it has been synced before), recording the full chunk hash list at the last sync time.

[0074] For a large file with new version:

[0075] First, quickly compute its H meta. Compare its H meta with the H meta of the last version of this file recorded in the cloud. If H meta is the same, it is very likely that the file content has not changed. At this time, the file can be considered "unmodified" and no content needs to be transmitted. H incremental directly uses the value of the last time. If H meta is different (such as Mtime changes), incremental hash calculation is needed:

[0076] Divide the file into fixed block size (such as BlockSize = 1MB or 4MB). Calculate the hash value of each block (H block1, H block2,..., H blockN). Combine these block hash values (such as sorted and concatenated) to form H incremental.

[0077] Final representation: The characteristics of a large file are represented by H meta and H incremental. H meta is used to quickly exclude unmodified files, and H incremental is used to accurately identify the changed data block.

[0078] Example:

[0079] A 10GB video file movie.mp4.

[0080] Initial upload: Calculate H_meta = SHA-1("movie.mp4" + "10737418240" + "1698765432" + "mp4") (assuming Size=10GB, Mtime=a timestamp). Then calculate the complete incremental block hash list (assuming a block size of 1MB, calculate the hash of 10240 blocks), obtaining H_incremental. Store (H_meta, H_incremental) in the cloud fingerprint database.

[0081] Second synchronization (file not modified): The client calculates the current H_meta and compares it with the H_meta of the file recorded in the cloud. If they are the same, the file is marked as "unmodified" and the upload is skipped.

[0082] Third synchronization (the user changed the filename to movie_final.mp4):

[0083] The client calculates the new H_meta = SHA-1("movie_final.mp4" + "10737418240" + "1698765432" + "mp4") (Mtime may or may not have changed). The new H_meta differs from the old H_meta recorded in the cloud (because the filename has changed). The client calculates the new complete H_incremental (a hash of 10240 blocks). The new (H_meta, H_incremental) is uploaded to the cloud for comparison. The cloud finds that the H_incremental is the same as the old H_incremental (although the filename has changed, the content remains the same).

[0084] Result: The cloud only needs to update the metadata (filename) of the file record and associate the new H_meta with the new H_incremental. There's no need to re-upload the 10GB file content. The client only uploads the metadata change value and the new hash signature.

[0085] S103. The client sends the hash feature value to the cloud server. The cloud server stores a hash fingerprint database. Each record in the hash fingerprint database contains a unique file identifier, file features, file physical storage address, and reference count. The file features are hash feature values ​​calculated by the corresponding file based on its size and using the corresponding hash calculation strategy.

[0086] This application establishes a hash fingerprint database in the cloud to store the hash characteristics of historical files. Before uploading a new file, the fingerprint database is queried, and if a match is found, the file address in the cloud is directly referenced.

[0087] Hash Fingerprint Database: a database stored in the cloud, each record contains at least:

[0088] File ID;

[0089] File Feature: H_full for small files, H_sample for medium files, (H_meta, H_incremental) H_incremental list itself for large files;

[0090] File Physical Storage Address (Cloud Storage Path);

[0091] Reference Count (how many user / file links point to this physical file).

[0092] S104, the cloud server queries the hash fingerprint database to determine whether there is a target record matching the hash feature value of the file to be synchronized;

[0093] S105, according to whether the target record exists in the hash fingerprint database, the synchronization of the file to be synchronized is completed.

[0094] Specifically, as shown in Figure 2 the synchronization flowchart of the file when the target record exists in the hash fingerprint database, including:

[0095] S201, the cloud server obtains the metadata of the file to be synchronized, and compares it with the file metadata corresponding to the target record, the metadata at least including one of file name, file size, file type, file last modification time, file creation time;

[0096] S202, if the metadata is completely consistent, the client establishes an association with the file physical storage address in the target record, the cloud server adds 1 to the reference count of the target record, and the synchronization of the file to be synchronized is completed;

[0097] S203, if the metadata is not completely consistent, it is determined as hash collision, the client uploads the file to be synchronized to the cloud server, the cloud server stores the file to be synchronized to a new physical storage address, and adds a new record in the hash fingerprint database, the file feature of the new record is the hash feature value of the file to be synchronized, the file physical storage address is the new physical storage address, and the reference count is set to 1, the synchronization of the file to be synchronized is completed.

[0098] Specifically, as shown in Figure 3 the synchronization flowchart of the file when the target record does not exist in the hash fingerprint database, including:

[0099] S301, the client uploads the file to be synchronized to the cloud server;

[0100] S302, the cloud server stores the file to be synchronized to a new physical storage address, and adds a record in the hash fingerprint library, the file characteristics of the added record are the hash characteristic value of the file to be synchronized, the file physical storage address is the new physical storage address, and the reference count is set to 1, completing the synchronization of the file to be synchronized.

[0101] For example:

[0102] 1. Client: The user prepares to upload a file photo.jpg (size 800KB, which is a small file).

[0103] 2. Client: Calculate H_full = SHA-1 (the entire photo.jpg content).

[0104] 3. Client to cloud: Send a query request containing the calculated H_full.

[0105] 4. Cloud: Find whether there is a record in the hash fingerprint library whose characteristic value is equal to H_full.

[0106] 5. Scene A: Match success (find the same H_full):

[0107] Cloud: Find a record whose H_full matches the value transmitted by the client. The record points to a physical file storage / abc123.jpg, and the reference count is 5 (indicating that there are 5 users or file links pointing to it).

[0108] Cloud to client: Response "match success", and return File ID and file physical storage address (storage / abc123.jpg).

[0109] Client: Do not upload the file content of photo.jpg.

[0110] Client: Link this photo.jpg to the File ID and file physical storage address found by the cloud in the user's file metadata record. This is equivalent to creating a "pointer" or "shortcut" in the user's file list pointing to the existing physical file in the cloud, and the reference count +1 (becomes 6).

[0111] Process connection: At this time, multi-dimensional conflict checking is performed. However, multi-dimensional conflict checking is performed locally on the client or in the cloud for metadata comparison, and no longer requires the transmission of file content. The purpose of the check is to prevent hash collisions from causing incorrect references.

[0112] Scenario B: Match failed (no same H_full found):

[0113] Cloud to client: Response "No match found".

[0114] Client: Perform full upload procedure (upload file content photo.jpg).

[0115] Cloud: Store the newly uploaded file at physical address storage / new_xyz789.jpg.

[0116] Cloud: Create a new record in the hash fingerprint library: File ID=NEW123, H_full= (just calculated hash), physical address=storage / new_xyz789.jpg, reference count=1.

[0117] Client: Record the user's photo.jpg link to File ID=NEW123.

[0118] Flow connection: After uploading, multi-dimensional conflict checking also needs to be performed, but this time it is checking whether the metadata and hash of the newly uploaded file are consistent.

[0119] Usage of cloud file address: The core role is that when multiple users upload files with the same content, the cloud only stores one physical file. All user file records point to the same physical data through File ID or directly through the physical storage address, which is the key to saving storage space.

[0120] Embodiment in subsequent steps: When users download files, the corresponding physical file is found according to the File ID or physical storage address in the user file record for downloading; when sharing files, the link (File ID) pointing to the physical file is shared; when deleting files, the reference count of the physical file is decremented, and when the reference count drops to 0, the physical file is actually deleted to release space. In the detection process, the address itself is returned to the client after a successful match, which is used by the client to establish a logical association between the local file and the cloud physical file. Subsequent file operations (reading, writing metadata, partial updating, deletion) are all located through this association to the correct physical file. In the checking process, the physical address does not need to be used to read the file content again, because a successful match means that the content hash is the same, and the checking is based on the metadata and the existing hash value. Double checking based on file metadata and hash value can avoid hash collision misjudgment.

[0121] The above double checking method is as follows:

[0122] Purpose: Relying on single hash value (even if sampling or chunking hash) only, there is a risk of hash collision (different content produces the same hash value), which can lead to false positive (different file is judged as the same) or false negative (the same file is judged as different). Introducing file metadata for double check can significantly reduce the false positive rate.

[0123] Primary comparison: Based on hash feature value (H_full, H_sample, or H_incremental), perform matching query in the cloud hash fingerprint library.

[0124] Secondary check after successful matching: When a hash match is found, do not immediately consider the file as a duplicate, but additionally compare the key metadata of the client file with the key metadata of the file corresponding to the matching record in the cloud. Commonly used metadata includes:

[0125] File size (Size): The most direct and effective check item. Different size of file content must be different (hash collision is theoretically possible, but different size is absolute evidence);

[0126] File type / extension (Type / Extension): Provides auxiliary information of content type;

[0127] File last modification time (Mtime);

[0128] File creation time (Ctime);

[0129] Check logic: If all the compared metadata items are completely consistent, it is finally confirmed that the file content is the same, and the reference operation is performed. If any of the compared metadata items is inconsistent (especially the file size is different), it is determined that hash collision occurs, the reference matching result is rejected, and the file is treated as a new file (upload content and create a new fingerprint record).

[0130] Example 1 (Prevent hash collision - size difference):

[0131] Client file A: report_v1.docx, size 1234567 bytes, calculate H_sample = ABC123 (assuming a medium file);

[0132] Cloud query: find a record H_sample = ABC123 in the fingerprint library, but its corresponding file size is 9876543 bytes (a completely different file, coincidentally the same sampling hash);

[0133] Double check: compare file size 1234567!= 9876543;

[0134] Result: Hash collision, reject reference. Client uploads report_v1.docx as a new file.

[0135] Example 2 (Prevent Hash Collision - Different Types):

[0136] Client File B: a malicious program disguised as image.jpg, actually.exe, size 500KB, compute H_full = DEF456 (small file);

[0137] Cloud Query: find a record H_full = DEF456 in the hash fingerprint library, which corresponds to a real picture file landscape.jpg, size is also 500KB, type jpeg;

[0138] Double Check:

[0139] Compare size: 500KB == 500KB (consistent);

[0140] Compare type / extension: client file claims to be.jpg, but cloud record type is jpeg.

[0141] Result: Inconsistent file header or real content type detection, determine as hash collision or file disguise, reject reference upload.

[0142] Example 3 (Normal Match - Metadata Consistent):

[0143] Client File C: budget.xlsx, size 2100000 bytes (medium file), last modified time 2023-10-27 10:30:00, compute H_sample = XYZ789;

[0144] Cloud Query: find a record H_sample = XYZ789 in the fingerprint library, whose corresponding file size is 2100000 bytes, type xlsx, last modified time 2023-10-27 10:30:00;

[0145] Double Check: size, type, modification time are consistent;

[0146] Result: Finally confirm that the file contents are the same. Client establishes a link, reference count +1. No need to upload file content.

[0147] Therefore, in some embodiments, the above-mentioned file synchronization method based on dynamic hash detection provided by the present application further comprises: when it is determined that there is a hash collision, after the client uploads the to-be-synchronized file to the cloud server, the cloud server re-computes the hash feature value of the uploaded to-be-synchronized file, and performs consistency checking with the hash feature value sent by the client, and after the consistency checking passes, the operation of adding a record in the hash fingerprint library is performed.

[0148] Therefore, in some embodiments, the above-mentioned file synchronization method based on dynamic hash detection provided by the present application further comprises: when a user performs a file deletion operation, the client sends a deletion request to the cloud server, the cloud server finds the target record corresponding to the file, reduces the reference count of the target record by 1, if the reference count is reduced to 0, the target record and the file in the corresponding file physical storage address are deleted; if the reference count is not reduced to 0, only the association between the user and the file physical storage address is deleted, and the target record and the file in the corresponding file physical storage address are retained.

[0149] In addition, in some embodiments, the above-mentioned file synchronization method based on dynamic hash detection provided by the present application further comprises: the cloud server periodically optimizes the hash fingerprint library, and deletes invalid records, the invalid records being records whose reference count is 0 or whose corresponding file in the file physical storage address has been deleted.

[0150] In addition, in some embodiments, the above-mentioned file synchronization method based on dynamic hash detection provided by the present application further comprises: the data transmission between the client and the cloud server adopts an encryption protocol, the encryption protocol being an SSL / TLS protocol, to ensure the security of the hash feature value, the metadata and the file content of the to-be-synchronized file in the transmission process.

[0151] The beneficial effects of the file synchronization method based on dynamic hash detection provided by the present application are as follows:

[0152] a) Hierarchical dynamic hash calculation method based on file size

[0153] Efficiency and accuracy dynamic balance: Differentiated hash strategies are adapted to the characteristics of files of different sizes to avoid the problems of "excessive calculation of small files" and "lagging calculation of large files". Small files are calculated by full hash to ensure detection accuracy, medium files are calculated by sampling hash to reduce the amount of calculation (for example, a 500MB PDF file is calculated by 32000 hashes, which reduces the amount of work by about 75%), and large files rely on metadata to quickly exclude unmodified files (for example, when a 10GB video is not modified, only H meta needs to be calculated to skip subsequent block hash, and the time consumption is shortened to milliseconds), which reduces the file hash calculation time by more than 60%, and greatly improves the synchronization efficiency of the network disk.

[0154] Efficient use of bandwidth resources: By using block hashing and incremental calculation to identify file sharing data blocks, redundant content transmission is avoided. For example, when different users upload video files containing the same segment, the system only needs to transmit the changed data blocks (such as a 10GB video file with a modified file name, only the new H_meta and H_incremental are uploaded, not the 10GB complete file). In the single file duplication synchronization scenario, the bandwidth consumption is reduced by more than 95%, reducing the cost of network disk operation and user synchronization waiting time.

[0155] Adapt to multiple scenarios of file processing: Covering the full-scenario needs from KB-level small files (such as documents, pictures) to GB-level large files (such as videos, compressed packages), without the need for manual intervention to adjust the strategy. Whether it is a personal user's daily document synchronization or a batch of large file transmission in an enterprise, the system can maintain high and stable hash calculation performance, improving the universality and practicality of the technical solution.

[0156] b) Dynamic updating of cloud hash fingerprint library and fast matching mechanism for duplicate files

[0157] Optimal use of storage resources: By using the "single physical file + multiple logical references" mode, the same file is avoided from being stored repeatedly. When multiple users upload the same content file (such as an enterprise team sharing the same template document), the cloud only keeps 1 copy of the physical data, and all users reference it through the File ID. In the single duplicate file scenario, the storage occupancy is reduced by (N-1) x 100% (N is the number of reference users), significantly reducing the storage pressure on the cloud and the cost of hardware investment.

[0158] Fast response to duplicate matching: Before uploading a new file, the system first queries the fingerprint library to quickly locate the existing file based on the hash characteristics (such as matching an 800KB image, which only takes milliseconds to query). There is no need to wait for the file to be completely uploaded before determining duplication. When a match is found, the content transmission is skipped, and the file synchronization time is shortened from "minutes" (such as large file upload) to "seconds" (only metadata and hash information are transmitted), improving user experience.

[0159] Intelligent and efficient data management: Relying on reference counting to realize dynamic recycling of physical files, when the file reference count drops to 0, the redundant physical data is automatically deleted, avoiding the occupation of storage space by invalid data; at the same time, the File ID is used to associate file lifecycle operations (download, share, delete), simplifying data management logic, reducing the complexity of cloud system operation and maintenance, and improving the accuracy and efficiency of data management.

[0160] c) Multi-dimensional conflict verification model combining hash values and file metadata

[0161] Greatly reduced risk of misjudgment: Through "hash matching + metadata verification" double protection, effectively avoid the misjudgment problem caused by single hash collision. For example, when different size files have hash collision (such as 123KB document and 987KB file have the same hash value), file size verification can directly determine that they are different files; when the file is disguised (such as.exe is disguised as.jpg), type verification can identify the abnormality, and the misjudgment rate is reduced to close to 0, ensuring the accuracy of user data transmission and storage.

[0162] Additional protection for data security: Metadata verification can identify file disguise and malicious tampering behavior. For example, when a malicious program is disguised as a normal picture file, through consistency verification of type, size and other metadata and hash characteristics, abnormal file reference upload can be rejected in time to avoid malicious files entering the cloud system, adding a layer of protection barrier for network disk data security and improving the security of the technical solution.

[0163] Lightweight running of verification logic: Metadata (such as size, type, modification time) acquisition and comparison takes very short time (milliseconds), and will not increase the additional burden of the system. While ensuring the accuracy of verification, it does not affect the overall efficiency of hash calculation and matching, realizing the dual goals of "high-precision verification + high-speed operation", balancing data accuracy and system performance.

[0164] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and description of the present application.

Claims

1. A method for detecting a network disk file synchronization based on dynamic hash, characterized in that, The method comprises the following steps: The client obtains a file to be synchronized, and determines a corresponding hash calculation strategy according to the size of the file to be synchronized; The client calculates a hash feature value of the file to be synchronized based on the determined hash calculation strategy; The client sends the hash feature value to a cloud server, wherein the cloud server stores a hash fingerprint library, and each record in the hash fingerprint library contains a file unique identifier, a file feature, a file physical storage address and a reference count; wherein the file feature is a hash feature value of the corresponding file calculated by the corresponding hash calculation strategy according to the size of the file; The cloud server queries the hash fingerprint library to determine whether there is a target record matching the hash feature value of the file to be synchronized; If the target record exists, the cloud server obtains the metadata of the file to be synchronized, and compares the metadata with the file metadata corresponding to the target record; the metadata at least includes one of the file name, the file size, the file type, the file last modification time and the file creation time; if the metadata is completely consistent, the client establishes an association with the file physical storage address in the target record, the cloud server adds 1 to the reference count of the target record, and the synchronization of the file to be synchronized is completed; if the metadata is not completely consistent, it is determined as hash collision, the client uploads the file to be synchronized to the cloud server, the cloud server stores the file to be synchronized to a new physical storage address, and adds a new record in the hash fingerprint library; the file feature of the new record is the hash feature value of the file to be synchronized, the file physical storage address is the new physical storage address, and the reference count is set to 1, and the synchronization of the file to be synchronized is completed; If the target record does not exist, the client uploads the file to be synchronized to the cloud server, the cloud server stores the file to be synchronized to a new physical storage address, and adds a new record in the hash fingerprint library; the file feature of the new record is the hash feature value of the file to be synchronized, the file physical storage address is the new physical storage address, and the reference count is set to 1, and the synchronization of the file to be synchronized is completed.

2. The method of claim 1, wherein, The method comprises the following steps: If the size of the file to be synchronized is less than 1MB, a full hash calculation strategy is determined; If the size of the file to be synchronized is greater than or equal to 1MB and less than or equal to 1GB, a block sampling hash calculation strategy is determined; If the size of the file to be synchronized is greater than 1GB, a calculation strategy combining metadata and incremental block hash is determined.

3. The method of claim 2, wherein, The full hash calculation strategy is that the client calculates a full hash value by using a preset hash function on the complete content of the file to be synchronized, and the full hash value is the hash feature value of the file to be synchronized.

4. The method of claim 2, wherein, The block sampling hash calculation strategy comprises the following steps: The client logically divides the file to be synchronized into data blocks of a fixed size to obtain the total number of blocks of the file to be synchronized; extracting data blocks in the front of the first preset proportion, data blocks in the middle of the second preset proportion and data blocks in the back of the third preset proportion in the file to be synchronized, wherein the number of the data blocks in the front of the first preset proportion a is the upward rounding of the product of a and the total number of blocks, the start offset of the data blocks in the middle of the second preset proportion b is equal to , the number of the data blocks in the back of the third preset proportion c is the upward rounding of the product of c and the total number of blocks, and the start offset of the data blocks in the back of the third preset proportion c is equal to . Size is the size of the file to be synchronized; The preset hash function is used to calculate the hash value of each extracted data block, the hash values of all extracted data blocks are spliced in sequence, and the preset hash function is used to calculate the hash value of the spliced result again to generate a sampling hash value, which is the hash characteristic value of the file to be synchronized.

5. The method of claim 2, wherein, The calculation strategy of the combined metadata and incremental block hash is: The client uses a preset hash function to hash calculate the metadata of the file to be synchronized to obtain a metadata hash value H_meta of the file to be synchronized. The file to be synchronized is divided into data blocks according to a fixed size, a preset hash function is used to calculate the hash value of each data block, and all hash values of the data blocks are combined to form an incremental block hash list H_incremental, and the hash characteristic value of the file to be synchronized is (H_meta, H_incremental).

6. The method of claim 1, wherein, The method further comprises: When it is determined that there is a hash collision, the client uploads the file to be synchronized to the cloud server, the cloud server re-calculates the hash characteristic value of the uploaded file to be synchronized, and performs consistency checking with the hash characteristic value sent by the client, and after the checking is passed, the operation of adding a record in the hash fingerprint library is performed.

7. The method of claim 1, wherein, The method further comprises: When the user performs a file deletion operation, the client sends a deletion request to the cloud server, the cloud server finds the target record corresponding to the file, reduces the reference count of the target record by 1, and if the reference count is reduced to 0, the target record and the file in the corresponding file physical storage address are deleted; if the reference count is not reduced to 0, only the association between the user and the file physical storage address is deleted, and the target record and the file in the corresponding file physical storage address are retained.

8. The method of claim 1, wherein, The method further comprises: The cloud server periodically optimizes the hash fingerprint library and deletes invalid records, the invalid records being records with a reference count of 0 or a file in a file physical storage address corresponding to the record having been deleted.

9. The method of claim 1, wherein, The method further comprises: The data transmission between the client and the cloud server adopts an encryption protocol, the encryption protocol being an SSL / TLS protocol, to ensure the security of the hash characteristic value, metadata and file content of the file to be synchronized in the transmission process.

Images