File-level duplicate removal method, system and device of multi-dimensional filtering structure

Abstract

The invention provides a file-level duplicate removal method, system and device of a multi-dimensional filtering structure. The method comprises the steps that a server side receives file fingerprints of files to be subjected to duplicate removal processing sent by all terminals; the file fingerprints sent by the terminals are judged through a multi-level duplicate removal mechanism, and duplicate removal judgment results of all levels are sent to the corresponding terminals; and the terminal executes the operation of continuously uploading or terminating the uploading of the file according to the judgment result. According to the scheme, high efficiency and low misjudgment rate of file deduplication in a high-concurrency scene are achieved in the whole process through hierarchical cooperation, the storage pressure of system files can be effectively relieved through the scheme, and the overall processing capacity of the system files is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a file-level deduplication method, system, and apparatus with a multi-dimensional filtering structure. Background Technology

[0002] With the acceleration of digitalization, the amount of data in storage systems is growing exponentially, making efficient use of storage resources and reducing data management costs core industry needs. File deduplication technology can efficiently optimize storage resources by accurately identifying duplicate data and eliminating redundancy. At the same time, it intervenes early in the data backup, archiving, and uploading stages, reducing invalid transmission and system burden, and improving data processing efficiency.

[0003] Currently, this technology has expanded from traditional storage systems (such as disk arrays and network-attached storage) to fields such as cloud computing, big data analysis, and information retrieval, but the need for performance optimization is becoming increasingly urgent as application scenarios become more diverse.

[0004] Current mainstream file deduplication solutions are based on single-level hash comparison, traditional cache hits, or single-dimensional Bloom filters. However, these solutions have the following key problems, making them difficult to adapt to high-concurrency uploads and large-scale distributed storage requirements:

[0005] Hash comparison is costly, and when dealing with a large number of large files, the time complexity is high, and the deduplication efficiency drops significantly.

[0006] Low cache hit rate; traditional single-level cache does not distinguish between hot and cold data characteristics, which exacerbates the overhead of repeated calculations.

[0007] One-dimensional Bloom filters have a high false positive rate and are prone to missing duplicate files or redundant writing.

[0008] With strong coupling to the database, the key data for deduplication depends on the backend database for storage and operation. In large-scale scenarios, the database can easily become a bottleneck, causing the deduplication process to be blocked and the response to be delayed.

[0009] In summary, existing technical solutions are insufficient in terms of efficiency, hit rate, false positive control, and coupling, and cannot meet the requirements of high concurrency and large-scale data deduplication. There is an urgent need to develop a file deduplication method with higher performance, lower false positives, and lower coupling to fill the technological gap. Summary of the Invention

[0010] The purpose of this application is to provide a file-level deduplication method, system, and apparatus with a multi-dimensional filtering structure to improve the real-time performance, accuracy, and resource utilization efficiency of data deduplication, effectively alleviate storage pressure, and improve the overall file processing capability of the system. The specific technical solution is as follows:

[0011] In a first aspect of this application, a file-level deduplication method with a multi-dimensional filtering structure is provided. The method is applied to a server and includes:

[0012] The server receives file fingerprints of files to be deduplicated from each terminal.

[0013] The server uses a multi-level deduplication mechanism to determine the file fingerprints sent by each terminal and sends the deduplication determination results at each level to the corresponding terminal.

[0014] This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

[0015] In an optional embodiment of this application, the server uses a multi-level deduplication mechanism to determine the file fingerprints sent by each terminal, and sends the deduplication determination results at each level to the terminal, including:

[0016] S1. Uploading Task Filtering Layer Determination: The server maintains a set of file fingerprints for each terminal's currently uploading tasks. If the file fingerprint exists in the set, the terminal is informed that the file is being uploaded, so that the terminal sorts the deduplication determination request for the file to the end of the queue and re-initiates the query after a preset time.

[0017] If the file fingerprint does not exist in the set, proceed to step S2;

[0018] S2, LRU numerical hot cache layer determination: Convert the file fingerprint into an unsigned numerical type and establish a mapping relationship between the file fingerprint and the unsigned numerical value; use a hash table with Least Recently Used (LRU) eviction policy and segmented lock characteristics to determine if the converted unsigned numerical value is duplicated; if a match is found, it is determined that the file is duplicated and the terminal is notified that the file does not need to be uploaded.

[0019] If no match is found, proceed to step S3;

[0020] S3. Multidimensional Bloom Anti-penetration Layer Determination: The file fingerprint is determined by a multidimensional Bloom filter, where each filter uses a different set of hash functions; if the determination result is "file does not exist", the terminal is informed that it can be uploaded;

[0021] If the determination result is "the file may exist", then proceed to step S4;

[0022] S4. Persistent data layer determination: The existence of the file fingerprint is verified by the storage backend; if it exists, it is determined that the file is duplicated, the terminal is notified that the file does not need to be uploaded, and the unsigned value corresponding to the file fingerprint is added to the LRU numerical hot cache layer.

[0023] If the file does not exist, inform the terminal to continue archiving and backing up or to upload the file.

[0024] The terminal performs corresponding file upload or archive backup operations based on the information provided by the server.

[0025] In an optional embodiment of this application, the task filtering layer determination during the upload process further includes:

[0026] Based on the estimated file upload time, determine whether the current file upload operation has timed out; if so, remove the file fingerprint corresponding to the task.

[0027] In an optional embodiment of this application, converting the file fingerprint into an unsigned numeric type includes:

[0028] If the file fingerprint is a hexadecimal string of length 32, extract the lower 0-15 bytes as the low-order string and extract the 16-31 bytes as the high-order string;

[0029] The high-order string and the low-order string are converted into 64-bit numbers respectively, and stored in the high 64-bit member and the low 64-bit member of the unsigned numeric type.

[0030] In an optional embodiment of this application, the LRU numerical hot cache layer determination further includes:

[0031] When the upload task is completed, the file fingerprint corresponding to the task is converted into an unsigned value according to the mapping rules and then added to the hot cache layer.

[0032] And when a deduplication request penetrates to the persistent data layer and hits, the unsigned value corresponding to the file fingerprint is added to the hot cache layer, thereby realizing the conversion of cold data to hot data.

[0033] In an optional embodiment of this application, the parameters of the multidimensional Bloom filter are configured as follows: the bit array length and the number of hash functions of the Bloom filter are adjusted according to the false positive rate requirements of the actual application scenario to constrain the false positive rate.

[0034] In one optional embodiment of this application, the server and the terminal communicate via Remote Procedure Call (RPC) to transmit data.

[0035] In a second aspect of this application, a file-level deduplication system with a multi-dimensional filtering structure is also provided, comprising: a server and one or more terminals; the terminals are used to calculate the file fingerprint of the file to be processed, and submit the file fingerprint to the server for deduplication operation via RPC remote procedure call;

[0036] The server is used to receive the file fingerprint and perform a deduplication operation on the file to be processed in accordance with any of the methods described in the first aspect.

[0037] In a third aspect of this application, a file-level deduplication apparatus with a multi-dimensional filtering structure is provided, applied on a server side, the apparatus comprising:

[0038] The receiving module is used to receive file fingerprints of files to be deduplicated, sent by each terminal.

[0039] The judgment and sending module is used to judge the file fingerprint sent by each terminal through a multi-level deduplication mechanism, and send the deduplication judgment results of each level to the terminal.

[0040] This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

[0041] In four aspects of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] A processor, when executing a program stored in memory, implements the method described in any one of the first aspects above.

[0044] In a fifth aspect of the embodiments of this application, a storage medium is also provided, the storage medium storing instructions that, when run on a computer, cause the computer to perform a file-level deduplication method with a multidimensional filtering structure as described in any of the first aspects above.

[0045] In a sixth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform a file-level deduplication method with a multidimensional filtering structure as described in any of the first aspects above.

[0046] The technical solution provided in this application embodiment involves the server receiving file fingerprints of files to be deduplicated from each terminal; then, it uses a multi-level deduplication mechanism to determine the file fingerprints sent by each terminal and sends the deduplication determination results of each level to the corresponding terminal; the terminal then performs the operation of continuing to upload or terminating the upload of the file based on the determination result. Furthermore, in this application solution, after the terminal obtains and calculates the file fingerprint, it submits it to the server, where the server's multi-level deduplication mechanism performs deduplication determination, and then the specific determination results of each level are used to perform the upload or terminate the upload operation. The entire process achieves high efficiency and low false positive rate for file deduplication in high-concurrency scenarios through hierarchical collaboration. This solution can effectively alleviate system file storage pressure and improve the overall file processing capacity of the system. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a file-level deduplication method with a multidimensional filtering structure provided in an embodiment of this application;

[0050] Figure 2 A schematic diagram of a file-level deduplication system with a multidimensional filtering structure provided in an embodiment of this application;

[0051] Figure 3 A schematic diagram of the deduplication determination process of a file-level deduplication method with a multidimensional filtering structure provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the data flow of a file-level deduplication method with a multi-dimensional filtering structure provided in an embodiment of this application.

[0053] Figure 5 This application provides a file-level deduplication device with a multi-dimensional filtering structure.

[0054] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0057] With the exponential growth of data volume in storage systems, efficient utilization of storage resources and reduction of data management costs have become core industry demands. This application provides a file-level deduplication method, system, and apparatus with a multi-dimensional filtering structure, which aims to solve the problem of low deduplication efficiency in the process of backing up or uploading a large amount of duplicate data in massive data environments such as cloud storage, backup servers, and archiving systems. It is particularly suitable for handling the problem of a large number of duplicate and redundant copies generated by frequent backups and repeated uploads.

[0058] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0059] like Figure 1 As shown, this application provides a file-level deduplication method with a multi-dimensional filtering structure, the method including the following steps S101-S102:

[0060] S100: The server receives file fingerprints of files to be deduplicated from each terminal.

[0061] S200: The file fingerprints sent by each terminal are determined through a multi-level deduplication mechanism, and the deduplication determination results at each level are sent to the corresponding terminal.

[0062] This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

[0063] The proposed solution significantly improves the real-time performance, accuracy, and resource utilization efficiency of data deduplication by integrating a multi-level deduplication mechanism and a refined filtering strategy. This effectively alleviates storage pressure and improves the overall system processing performance, making it suitable for high-performance file-level deduplication in scenarios such as cloud storage, data archiving, and file upload.

[0064] In one embodiment of this application, in step S102 above, the server determines the file fingerprints sent by each terminal through a multi-level deduplication mechanism, and sends the deduplication determination results of each level to the terminal, specifically including the following steps S1-S4:

[0065] S1. Uploading Task Filtering Layer Determination: The server maintains a set of file fingerprints for each terminal's currently uploading tasks. If the file fingerprint exists in the set, the terminal is informed that the file is being uploaded, so that the terminal sorts the deduplication determination request for the file to the end of the queue and re-initiates the query after a preset time.

[0066] If the file fingerprint does not exist in the set, proceed to step S2;

[0067] S2, LRU numerical hot cache layer determination: Convert the file fingerprint into an unsigned numerical type and establish a mapping relationship between the file fingerprint and the unsigned numerical value; use a hash table with Least Recently Used (LRU) eviction policy and segmented lock characteristics to determine if the converted unsigned numerical value is duplicated; if a match is found, it is determined that the file is duplicated and the terminal is notified that the file does not need to be uploaded.

[0068] If no match is found, proceed to step S3;

[0069] S3. Multidimensional Bloom Anti-penetration Layer Determination: The file fingerprint is determined by a multidimensional Bloom filter, where each filter uses a different set of hash functions; if the determination result is "file does not exist", the terminal is informed that it can be uploaded;

[0070] If the determination result is "the file may exist", then proceed to step S4;

[0071] S4. Persistent data layer determination: The existence of the file fingerprint is verified by the storage backend; if it exists, it is determined that the file is duplicated, the terminal is notified that the file does not need to be uploaded, and the unsigned value corresponding to the file fingerprint is added to the LRU numerical hot cache layer.

[0072] If the file does not exist, inform the terminal to continue archiving and backing up or to upload the file.

[0073] The aforementioned terminals perform corresponding file upload or archive backup operations based on the information provided by the server.

[0074] See details Figure 2, Figure 3 and Figure 4 The illustrated embodiment shows that in this solution, the terminal first calculates the file hash value (i.e., file fingerprint), and then sends a file deduplication request to the server via an RPC call module. After the server performs deduplication determination on the file fingerprint, it sends a deduplication response to the terminal. The server's multi-level file deduplication determination mechanism mainly consists of the following four levels:

[0075] First level: Upload task filtering layer

[0076] This layer maintains the currently uploading tasks for each terminal. When a terminal passes the deduplication check via RPC (the file does not exist in the storage system) and begins uploading, it reports the fingerprint of the file to be uploaded. A notification is also sent upon completion of the upload. This layer is primarily used to improve deduplication accuracy, filter files currently being uploaded, prevent duplicate upload requests within a short period, and eliminate the problem of insufficient deduplication accuracy caused by the time difference between numerous terminals simultaneously uploading the same file.

[0077] In one possible embodiment of this application, the task filtering layer determination during the upload process further includes:

[0078] Based on the estimated file upload time, determine whether the current file upload operation has timed out; if so, remove the file fingerprint corresponding to the task.

[0079] This level estimates the upload time based on file size and registers a timeout culling plan based on the estimation results to prevent tasks from being suspended at this processing level for a long time due to terminal crashes, abnormal terminations, or other reasons.

[0080] In this embodiment, if the reported file fingerprint does not exist in this layer, it needs to be further determined by the device at the next level. If it does not exist in this layer, the terminal is replied to that it is uploading. In one possible design, the terminal that receives the reply knows that other terminals are uploading the file. Since it is uncertain whether other terminals can eventually archive or upload the file successfully based on the current status, the file can be sorted to the end of the deduplication determination queue. After a certain period of time, the deduplication RCP query will continue until a reply confirming whether the file is in the system is obtained.

[0081] Second level: LRU numerical hot cache layer

[0082] The file fingerprint is a unique file ID calculated by the client using a general hash algorithm. This ID is a string composed of hexadecimal characters 0-f. This level converts the reported file fingerprint into an unsigned numeric type, which in one possible design could be uint128, uint256, etc., and establishes a corresponding mapping relationship between the file fingerprint and the unsigned numeric type to meet the deduplication requirements of different levels in the future.

[0083] Taking uint128 as an example, it mainly defines two uint64 members to store the high 64 bits and the low 64 bits of data respectively. It can also be seen that since the computer world counts from 0, the maximum value that a 64-bit binary number can represent is 2^64 - 1 = 0xffff ffff ffff ffff. That is, the maximum length of the low 64 bits and the high 64 bits mapped to the original hexadecimal string of the file fingerprint is 16.

[0084] The specific conversion process is as follows: For example, if the reported file fingerprint FECDFE418E804A7E49011EBAAA24C38D is a string of length 32, the file fingerprint is truncated from the low 0-15 bytes to the low byte 49011EBAAA24C38D, and from the 16-31 bytes to the high byte FECDFE418E804A7E. Then, the high and low bytes are converted into the actual stored numerical type using the following formula (1).

[0085] Result= Formula (1)

[0086] Where i = 0 is the least significant bit (rightmost bit), and di is the decimal value of the i-th bit, 16 i It is the weight of that position.

[0087] This layer module mainly implements a hash table with LRU (Least Recently Used) eviction policy and segmented lock characteristics; the pre-static declaration states that the hash table consists of m hash slots, and all slots are sequentially divided into n segments, each segment is controlled by an independent lock, and combined with the direct mapping relationship of numerical types, the hash bucket hit determination is accelerated.

[0088] As a hot cache layer, it needs to work closely with other modules and devices at the upper and lower levels. For example, when the upload task is completed, the corresponding file fingerprint will be added to the hot cache layer according to the mapping rules. When the deduplication request penetrates to the persistent data layer and hits, it is necessary to convert the cold data into hot data (i.e., add it to the hot cache layer).

[0089] If the reported file fingerprint exists at this level, it is determined to be a duplicate file in the system and a reply is sent to the terminal. The terminal does not need to archive or upload the file. Otherwise, it proceeds to the next level of device for further evaluation.

[0090] Third layer: Multi-dimensional bleed-proof layer

[0091] This layer consists of a multidimensional Bloom filter. In one possible implementation, deduplication can be performed based on different parts of the file fingerprint, with each filter using a different set of hash functions to improve the reliability of deduplication. Meanwhile, as those skilled in the field know, the Bloom structure is a data structure based on a mathematical probability model with extremely high space utilization. However, it has a certain false positive rate for existing events due to space collisions. Users need to adjust parameters such as the Bloom space length and the number of hashes according to the usage scenario to constrain the false positive rate within a tolerable range for the system. However, as a precise system, to avoid false positives, unlike the hit events of the preceding devices, this layer only retains confident non-hit events, indicating to the terminal that the file does not exist. This is mainly to avoid database penetration. If the file fingerprint returns "possibly exists" when passing through this layer's device, it must proceed to the next layer's device for further judgment.

[0092] Therefore, the parameter configuration method of the above-mentioned multidimensional Bloom filter is as follows: adjust the bit array length and the number of hash functions of the Bloom filter according to the false positive rate requirements of the actual application scenario to constrain the false positive rate.

[0093] Fourth layer: Persistent data layer

[0094] When a file fails to pass through the upload task, cache, and Bloom filter, it needs to enter the persistence layer to verify its existence through the storage backend. If the file exists in the storage backend, it will be considered a duplicate file in the system, and the terminal will be informed that the file exists. The data will also be promoted from cold data to hot data and added to the second-level hot cache layer to accelerate deduplication. Otherwise, the terminal will be directly informed that the file does not exist, and the terminal needs to continue with the archiving or uploading of this task in the system.

[0095] Reference Figure 2 The embodiment shown in this solution provides a file-level deduplication system with a multi-dimensional filtering structure, including a server and one or more terminals.

[0096] The terminal device has a client installed. The terminal includes a file scanning module and an RCP module. The file scanning module is used to calculate the file fingerprint of the file to be processed, and the RCP module submits the file fingerprint to the server for deduplication via RPC remote procedure call.

[0097] The terminal sends a file deduplication request to the server, which carries the corresponding file fingerprint. The server receives the file fingerprint and performs deduplication determination through a multi-level file deduplication device. This multi-level deduplication mechanism includes the aforementioned uploading task filtering layer, LRU numerical hot cache layer, multidimensional Bloom anti-penetration layer, and persistent cold data layer. In the file deduplication process, the uploading task filtering layer first excludes dangling tasks, then the LRU hot cache layer quickly determines hot data duplication, then the multidimensional Bloom filter prevents database penetration, and finally the persistent data layer verifies cold data and updates the hot cache.

[0098] For details on the specific processing mechanisms of each layer, please refer to the description of the above method embodiments, which will not be repeated here.

[0099] See Figure 5 The embodiment shown provides a file-level deduplication device with a multi-dimensional filtering structure, applied on a server side. The device includes:

[0100] The receiving module 501 is used to receive the file fingerprint of the file to be deduplicated sent by each terminal;

[0101] The determination and sending module 502 is used to determine the file fingerprint sent by each terminal through a multi-level deduplication mechanism, and send the deduplication determination results of each level to the terminal.

[0102] This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

[0103] This application also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0104] Memory 603 is used to store computer programs;

[0105] In one embodiment of this application, when the processor 601 executes the program stored in the memory 603, it performs the following steps: the server receives file fingerprints of files to be deduplicated from each terminal; the server determines the file fingerprints sent by each terminal through a multi-level deduplication mechanism, and sends the deduplication determination results of each level to the corresponding terminal; so that the terminal performs the operation of continuing to upload the file or terminating the upload according to the determination results.

[0106] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0107] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0108] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0109] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0110] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute a file-level deduplication method with a multidimensional filtering structure as described in any of the above embodiments.

[0111] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform a file-level deduplication method with a multidimensional filtering structure as described in any of the above embodiments.

[0112] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0113] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A file-level deduplication method with a multi-dimensional filtering structure, characterized in that, The method is applied to the server side, and the method includes: The server receives file fingerprints of files to be deduplicated from each terminal. The file fingerprints sent by each terminal are judged by a multi-level deduplication mechanism, and the deduplication judgment results at each level are sent to the corresponding terminal. This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

2. The method according to claim 1, characterized in that, The server uses a multi-level deduplication mechanism to determine the file fingerprints sent by each terminal, and sends the deduplication determination results at each level to the terminal, including: S1. Uploading Task Filtering Layer Determination: The server maintains a set of file fingerprints for each terminal's currently uploading tasks. If the file fingerprint exists in the set, the terminal is informed that the file is being uploaded, so that the terminal sorts the deduplication determination request for the file to the end of the queue and re-initiates the query after a preset time. If the file fingerprint does not exist in the set, proceed to step S2; S2, LRU numerical hot cache layer determination: Convert the file fingerprint into an unsigned numerical type and establish a mapping relationship between the file fingerprint and the unsigned numerical value; use a hash table with Least Recently Used (LRU) eviction policy and segmented lock characteristics to determine if the converted unsigned numerical value is duplicated; if a match is found, it is determined that the file is duplicated and the terminal is notified that the file does not need to be uploaded. If no match is found, proceed to step S3; S3. Multidimensional Bloom Anti-penetration Layer Determination: The file fingerprint is determined by a multidimensional Bloom filter, where each filter uses a different set of hash functions; if the determination result is "file does not exist", the terminal is informed that it can be uploaded; If the determination result is "the file may exist", then proceed to step S4; S4. Persistent data layer determination: The existence of the file fingerprint is verified by the storage backend; if it exists, it is determined that the file is duplicated, the terminal is notified that the file does not need to be uploaded, and the unsigned value corresponding to the file fingerprint is added to the LRU numerical hot cache layer. If the file does not exist, inform the terminal to continue archiving and backing up or to upload the file. The terminal performs corresponding file upload or archive backup operations based on the information provided by the server.

3. The method according to claim 1 or 2, characterized in that, The task filtering layer determination during the upload process also includes: Based on the estimated file upload time, determine whether the current file upload operation has timed out; if so, remove the file fingerprint corresponding to the task.

4. The method according to claim 1 or 2, characterized in that, The step of converting the file fingerprint into an unsigned numeric type includes: If the file fingerprint is a hexadecimal string of length 32, extract the lower 0-15 bytes as the low-order string and extract the 16-31 bytes as the high-order string; The high-order string and the low-order string are converted into 64-bit numbers respectively, and stored in the high 64-bit member and the low 64-bit member of the unsigned numeric type.

5. The method according to claim 1 or 2, characterized in that, The LRU numerical hot cache layer determination also includes: When the upload task is completed, the file fingerprint corresponding to the task is converted into an unsigned value according to the mapping rules and then added to the hot cache layer. And when a deduplication request penetrates to the persistent data layer and hits, the unsigned value corresponding to the file fingerprint is added to the hot cache layer, thereby realizing the conversion of cold data to hot data.

6. The method according to claim 1 or 2, characterized in that, The parameter configuration method of the multidimensional Bloom filter is as follows: adjust the bit array length and the number of hash functions of the Bloom filter according to the false positive rate requirements of the actual application scenario to constrain the false positive rate.

7. The method according to claim 1, characterized in that, The server and the terminal communicate via Remote Procedure Call (RPC) to transmit data.

8. A file-level deduplication system with a multi-dimensional filtering structure, characterized in that, include: Server and one or more terminals; The terminal is used to calculate the file fingerprint of the file to be processed, and submit the file fingerprint to the server for deduplication operation via RPC remote procedure call. The server is used to receive the file fingerprint and perform deduplication on the file to be processed in accordance with the method described in any one of claims 1-7.

9. A file-level deduplication device with a multi-dimensional filtering structure, characterized in that, Applied to the server side, the device includes: The receiving module is used to receive file fingerprints of files to be deduplicated, sent by each terminal. The judgment and sending module is used to judge the file fingerprint sent by each terminal through a multi-level deduplication mechanism, and send the deduplication judgment results of each level to the terminal. This allows the terminal to perform either continue uploading the file or terminate the upload based on the determination result.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

Images