Data storage method and apparatus for data deduplication service
By combining the seeding-period offline mode and the non-seeding-period online mode in the data deduplication business, and combining coarse and fine-grained deduplication strategies, the problem of high resource consumption on both the client and server sides is solved, and efficient processing of deduplication business is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUNINFO INFORMATION TECH
- Filing Date
- 2023-02-20
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, the efficiency bottleneck of data deduplication services mainly lies in the high resource consumption and long processing time on both the client and server sides, resulting in low efficiency of traditional deduplication seeding.
A method combining offline mode during the seeding period and online mode during the non-seeding period is adopted. During the seeding period, the deduplication client sends the data to be backed up to the server for deduplication. After the offline mode ends, it enters online mode to process the data in parallel with the server. The combination of coarse-grained and fine-grained deduplication strategies optimizes resource utilization.
This approach significantly improves the overall efficiency of deduplication operations, reduces resource consumption, and increases processing speed.
Smart Images
Figure CN116126817B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data storage technology for data deduplication operations. Background Technology
[0002] As the amount of data generated increases, the amount of duplicate data stored in the same storage cluster also increases. Therefore, if duplicate data can be eliminated and only a single copy is stored, the savings in storage costs will be significant. Data deduplication, or data deduplication, is precisely designed to achieve this.
[0003] In existing technologies, the high resource consumption and long processing time on both the client and server sides during the initial deduplication process make the traditional deduplication seeding period a bottleneck for the overall deduplication efficiency. Therefore, overcoming this bottleneck is a pressing issue. Summary of the Invention
[0004] The purpose of this application is to provide a data storage method and device for data deduplication services.
[0005] According to one aspect of this application, a data storage method for data deduplication services is provided, wherein the method includes:
[0006] After entering the offline mode during the seeding period, the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period.
[0007] The deduplication server performs a deduplication operation based on the first data to be backed up, in order to complete the data storage during the deduplication seeding period;
[0008] After the offline mode of the deduplication client during the seeding period ends, the deduplication client enters the online mode of the deduplication non-seeding period and performs the deduplication operation based on the second backup data to achieve parallel processing with the deduplication seeding period of the deduplication server.
[0009] The deduplication server performs data storage based on the data sent after the deduplication client performs the deduplication operation, thereby completing the data storage during the non-seeding period of deduplication.
[0010] Optionally, the method further includes:
[0011] The system determines whether the first data to be backed up is unbacked up based on a preset deduplication granularity level, wherein the preset deduplication granularity level is file-level granularity.
[0012] Specifically, after entering the offline mode during the seeding period, the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period, which includes: when the first backup data is unbacked up, after entering the offline mode during the seeding period, the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period.
[0013] Optionally, the deduplication operation includes:
[0014] Based on local device resource usage, different preset deduplication granularity levels are adaptively selected to perform deduplication operations.
[0015] Optionally, the deduplication operation includes:
[0016] The deleting operation is performed based on different file types, wherein the file types include at least one of images, videos, mirrors, and compressed files.
[0017] Furthermore, the deduplication operation includes data segmentation, fingerprint identification, and fingerprint deduplication.
[0018] According to another aspect of this application, a computer-readable medium is also provided, on which computer-readable instructions are stored, which can be executed by a processor to perform the operation of the aforementioned method.
[0019] According to another aspect of this application, a data storage device for a data deduplication service is also provided, wherein the device includes:
[0020] One or more processors; and
[0021] A memory storing computer-readable instructions, which, when executed, cause the processor to perform operations as described above.
[0022] Compared to existing technologies, this application, after entering the offline mode during the seeding period, allows the deduplication client to send the first batch of backup data to be deduplicated to the deduplication server during the seeding period. The deduplication server then performs the deduplication operation based on this first batch of backup data to complete the data storage for the seeding period. After the offline mode of the seeding period ends, the deduplication client enters the online mode of the non-seeding period and performs the deduplication operation based on the second batch of backup data, achieving parallel processing with the deduplication server's seeding period. The deduplication server then stores the data based on the data sent after the deduplication client's deduplication operation to complete the data storage for the non-seeding period. This optimized scheme, combining offline seeding with online non-seeding, effectively improves the overall efficiency of the deduplication service. Attached Figure Description
[0023] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0024] Figure 1 A flowchart illustrating a data storage method for a data deduplication service according to one aspect of this application is shown.
[0025] Figure 2 This diagram illustrates a data storage method for a data deduplication service according to a preferred embodiment of the present application.
[0026] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0027] The present invention will now be described in further detail with reference to the accompanying drawings.
[0028] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0029] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0030] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.
[0031] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.
[0032] Figure 1 This application illustrates a data storage method for data deduplication services, comprising:
[0033] After S11 enters the offline mode of seeding period, the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period.
[0034] The deduplication server described in S12 performs a deduplication operation based on the first data to be backed up, so as to complete the data storage during the deduplication seeding period;
[0035] S13 After the offline mode of the seeding period of the deduplication client ends, the deduplication client enters the online mode of the non-seeding period of deduplication and performs deduplication operation based on the second backup data to achieve parallel processing with the seeding period of the deduplication server.
[0036] The deduplication server described in S14 stores data based on the data sent after the deduplication client performs the deduplication operation, thereby completing the data storage during the non-seeding period of deduplication.
[0037] In this embodiment, in step S11, the deduplication seeding period refers to the first task execution of the same data backup path or backup strategy. That is, the process of converting the client's data to be backed up into data with a unique fingerprint database on the server side is called the deduplication seeding period. The offline mode means that the deduplication operation and deduplication data storage are asynchronously separated. In terms of overall architecture and operating model, both the deduplication client and the deduplication server in this scheme generally have the ability to segment data. Here, the first data to be backed up is the data that needs to be deduplicated. After entering the seeding period offline mode, the deduplication client will send the first data to be backed up to the deduplication server.
[0038] Preferably, prior to step S11, the method further includes:
[0039] The system determines whether the first data to be backed up is unbacked up based on a preset deduplication granularity level, wherein the preset deduplication granularity level is file-level granularity.
[0040] Step S11 includes: when the first data to be backed up is unbacked up and the offline mode of the seeding period is entered, the deduplication client sends the first data to be backed up to be deduplicated to the deduplication server during the deduplication seeding period.
[0041] Specifically, in this embodiment, a hierarchical strategy is adopted, using coarse-grained deduplication and identification methods with lower resource overhead (such as file-level deduplication) as a preparatory operation for parallel processing. This can significantly reduce the resource overhead of directly using fine-grained deduplication methods (such as block-level deduplication) in this step and greatly improve the running speed of the pre-operation.
[0042] By using a coarse-grained file-level identification method, it is determined whether the data to be backed up (e.g., the first data to be backed up) already has a stored backup record. If the coarse-grained identification finds that a backup record for the file already exists, subsequent backup operations for that data will not be performed, and the process will quickly proceed to the next backup operation. If the coarse-grained identification finds that the data to be backed up does not have any fingerprint information or related data on the server side, the data to be backed up is marked as unbacked up. The deduplication offline module will then perform fine-grained deduplication and fingerprint storage on all data items marked as unbacked up when the opportunity arises.
[0043] In a preferred embodiment, if a backup task with the same data content is executed twice consecutively, and some data from the first backup task has not yet been deduplicated by the offline deduplication module, the system first queries the coarse-grained deduplication list to determine if there is a matching coarse-grained identification identifier in the list. If not, it inserts the identifier into the deduplication list; otherwise, it ignores the identical backup data items in the current backup task. This method, as a preparatory retrieval operation before the block-level fingerprint database indexing, can effectively improve retrieval speed. For example, if the file fingerprint of the current backup file already exists in the coarse-grained identification list, then subsequent block-level fingerprint database indexing is unnecessary.
[0044] Continuing in this embodiment, in step S12, the deduplication server performs a deduplication operation based on the first data to be backed up to complete the data storage during the deduplication seeding period. Preferably, the deduplication operation includes data segmentation, fingerprint identification, and fingerprint deduplication. Here, after performing the deduplication operation, the deduplication server realizes the data storage after deduplicating the first data to be backed up.
[0045] Continuing in this embodiment, in step S13, after the offline mode of the deduplication client's seeding period ends, the deduplication client enters the online mode of the deduplication non-seeding period, performing deduplication operations based on the second data to be backed up, thereby achieving parallel processing with the seeding period of the deduplication server. Specifically, after the deduplication client sends the first data to be backed up, the offline mode of the deduplication client's seeding period can end, and it can immediately enter the online mode of the deduplication non-seeding period. Here, the deduplication non-seeding period refers to the period when the same backup strategy is used to perform a backup operation again, which is considered to be the non-seeding period. That is, the process of backing up the data content related to the backup that has been saved on the server side is the deduplication non-seeding period. Among them, the online mode means that the deduplication operation and the deduplication data storage are carried out simultaneously. In terms of overall architecture and operation model, the deduplication client of this scheme generally has the ability to segment data, while the deduplication server focuses more on fingerprint identification and data storage related operations. By combining offline operations during the seeding period with online operations during the non-seeding period in this way, parallel processing of the deduplication client and the deduplication server during the deduplication seeding period can be achieved, thereby effectively improving the overall efficiency of the deduplication business.
[0046] Continuing in this embodiment, in step S14, the deduplication server performs data storage based on the data sent after the deduplication client performs the deduplication operation, thereby completing the data storage of the non-seeding period of deduplication.
[0047] Preferably, the deduplication operation includes: adaptively selecting different preset deduplication granularity levels to perform the deduplication operation based on local device resource usage.
[0048] In this embodiment, different deduplication granularity levels can be selected to perform deduplication operations based on the resource usage of the local device. In this way, the maximum utilization of local resources can be achieved and the efficiency of deduplication operations can be improved.
[0049] Preferably, the deduplication operation further includes performing deduplication based on different file types, wherein the file types include, but are not limited to, at least one of images, videos, mirror images, and compressed files.
[0050] In this embodiment, multiple deduplication strategies are employed to address different deduplication application scenarios. These include classifications such as structured versus unstructured data, large versus fragmented files, and compressed versus uncompressed data. For different deduplication backup scenarios, a suitable deduplication strategy can be selected based on the characteristics and implementation features of different deduplication technologies. For example, a coarse-grained file-level deduplication strategy is more suitable for backup files exceeding 500MB, while a finer-grained block-level deduplication strategy can be used for backup files and data smaller than this preset threshold.
[0051] In a preferred embodiment, for high-compression file data sources, a larger block range results in a higher deduplication rate and faster backup speed. The deduplication block range can be set to 256–512KB, for example, for file types such as Word, TXT, and Log. For low-compression file data sources, when the block range is set to the default value, the deduplication rate is relatively high, and the backup speed is also relatively high. The deduplication block range can be set to 16–128KB, for example, for file types such as images, videos, mirrors, and compressed files. For large file data sources, file-level deduplication is recommended. For non-first-time backup scenarios, the backup speed is faster, and the minimum deduplication file size is preferably set to 500MB. For massive small file data sources, with a well-developed deduplication granularity strategy, volume-level deduplication is preferred; for typical deduplication implementations, file-level deduplication can be used to process massive small files. The above settings are merely examples and are not specific limitations; adjustments can be made based on different application scenarios.
[0052] Compared to existing technologies, this application, after entering the offline mode during the seeding period, allows the deduplication client to send the first batch of backup data to be deduplicated to the deduplication server during the seeding period. The deduplication server then performs the deduplication operation based on this first batch of backup data to complete the data storage for the seeding period. After the offline mode of the seeding period ends, the deduplication client enters the online mode of the non-seeding period and performs the deduplication operation based on the second batch of backup data, achieving parallel processing with the deduplication server's seeding period. The deduplication server then stores the data based on the data sent after the deduplication client's deduplication operation to complete the data storage for the non-seeding period. This optimized scheme, combining offline seeding with online non-seeding, effectively improves the overall efficiency of the deduplication service.
[0053] Figure 2 This diagram illustrates a data storage method for data deduplication according to a preferred embodiment of this application. The source end is a deduplication client, and the storage end is a deduplication server. The source data is first backup data. After the source data is sent, the offline mode of the seeding period ends, and the source end can enter the online mode of the deduplication non-seeding period to perform source data segmentation, thereby achieving parallel processing with the deduplication seeding period of the deduplication server.
[0054] According to another aspect of this application, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement the aforementioned method.
[0055] According to another aspect of this application, a data storage device for data deduplication services is also provided, wherein the device includes:
[0056] One or more processors; and
[0057] A memory storing computer-readable instructions, which, when executed, cause the processor to perform operations as described above.
[0058] For example, when a computer-readable instruction is executed, it causes one or more processors to: enter a seeding period offline mode, and the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period;
[0059] The deduplication server performs a deduplication operation based on the first data to be backed up, in order to complete the data storage during the deduplication seeding period;
[0060] After the offline mode of the deduplication client during the seeding period ends, the deduplication client enters the online mode of the deduplication non-seeding period and performs the deduplication operation based on the second backup data to achieve parallel processing with the deduplication seeding period of the deduplication server.
[0061] The deduplication server performs data storage based on the data sent after the deduplication client performs the deduplication operation, thereby completing the data storage during the non-seeding period of deduplication.
[0062] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
Claims
1. A data storage method for data deduplication service, wherein, The method includes: After entering the offline mode of the seeding period, the deduplication client sends the first data to be backed up for deduplication to the deduplication server during the deduplication seeding period. The offline mode means that the deduplication operation and the deduplication data storage are performed asynchronously and separately. The deduplication seeding period refers to the first task execution of the same data backup path or backup strategy. The deduplication server performs a deduplication operation based on the first data to be backed up, in order to complete the data storage during the deduplication seeding period; After the offline mode of the seeding period of the deduplication client ends, the deduplication client enters the online mode of the deduplication non-seeding period and performs deduplication operation based on the second data to be backed up, so as to achieve parallel processing with the deduplication seeding period of the deduplication server. The deduplication non-seeding period refers to the same backup strategy being used to perform backup operation again, and the online mode refers to the deduplication operation and deduplication data storage being carried out simultaneously. The deduplication server performs data storage based on the data sent after the deduplication client performs the deduplication operation, thereby completing the data storage during the non-seeding period of deduplication.
2. The method of claim 1, wherein, After entering the offline mode during the seeding period, before the deduplication client sends the first backup data to be deduplicated to the deduplication server during the deduplication seeding period, the method further includes: The system determines whether the first data to be backed up is unbacked up based on a preset deduplication granularity level, wherein the preset deduplication granularity level is file-level granularity. The step of sending the first backup data to be re-deleted to the deleting server during the deleting seeding period after entering the offline mode of the seeding period includes: when the first backup data is unbacked up, after entering the offline mode of the seeding period, the deleting client sends the first backup data to be re-deleted to the deleting server during the deleting seeding period.
3. The method of claim 1 or 2, wherein, The deduplication operation includes: Based on local device resource usage, different preset deduplication granularity levels are adaptively selected to perform deduplication operations.
4. The method of claim 1, wherein, The deduplication operation includes: The deleting operation is performed based on different file types, wherein the file types include at least one of images, videos, mirrors, and compressed files.
5. The method of claim 1, wherein, The deduplication operation includes data segmentation, fingerprint identification, and fingerprint deduplication.
6. A computer-readable medium having stored thereon computer-readable instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 5.
7. A data storage device for a data deduplication service, wherein, The device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the operations of the method as described in any one of claims 1 to 5.