Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device, a storage system is provided with a cloud storage node and a local storage node, and the method comprises the steps that in response to a data deduplication request, deduplication operation of multiple granularities is executed on the storage system; during system-level deduplication, deduplication is carried out on the data in the local storage node; during application-level deduplication, performing deduplication on non-local data in the same data accessed by different applications; during data block-level deduplication, deduplication is carried out on the data blocks in the cloud storage nodes; and during byte-level deduplication, deduplication is carried out on byte fragments in the data blocks of the storage system. By means of the data deduplication method and device, data deduplication can be conducted on the storage system from multiple granularities, and the data deduplication rate is increased.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data processing method and device. Background Art

[0002] With the explosive growth of data scale, data of different protocol types are also increasing. Deduplication technology can be used to delete duplicate data in storage systems, thereby improving storage system performance, reducing operating costs and improving data management efficiency. However, the existing deduplication technology has limited effect on deduplication and is difficult to meet the needs. Summary of the invention

[0003] In view of the above problems, a data processing method and device are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A data processing method, wherein a storage system is provided with a cloud storage node and a local storage node, and the method comprises:

[0005] In response to a data deduplication request, performing deduplication operations of multiple granularities on the storage system;

[0006] During system-level deduplication, deduplication is performed on the data in the local storage node;

[0007] During application-level deduplication, deduplication is performed on non-local data in the same data accessed by different applications;

[0008] In case of data block level deduplication, deduplication is performed on the data blocks in the cloud storage node;

[0009] In byte-level deduplication, byte segments in data blocks of the storage system are deduplicated.

[0010] Optionally, the deduplication operations of the multiple granularities have a hierarchical relationship, and the method further includes:

[0011] When executing a deduplication operation at the current level, determining whether the to-be-deleted data corresponding to the deduplication operation at the current level is within the deduplication data range of the deduplication operation at the previous level;

[0012] If the data to be deleted is within the deduplication data range of the deduplication operation at the previous level, then the deduplication operation on the data to be deleted at the current level is stopped;

[0013] If the data to be deleted is outside the deduplication data range of the deduplication operation at the previous level, a deduplication operation is performed on the data to be deleted at the current level.

[0014] Optionally, the hierarchical relationship between the deduplication operations of the multiple granularities is as follows from the upper layer to the lower layer:

[0015] System-level deduplication, application-level deduplication, data block-level deduplication, and byte-level deduplication.

[0016] Optionally, during the system-level deduplication, deduplication of data in the local storage node includes:

[0017] During system-level deduplication, determining the access frequency of data in the local storage node;

[0018] In the local storage node, deduplication is performed on the data whose access frequency is less than a frequency threshold.

[0019] Optionally, in the local storage node, deduplicating the data whose access frequency is less than a frequency threshold includes:

[0020] Determining a synchronization state of the data whose access frequency is less than a frequency threshold;

[0021] If the data whose access frequency is less than the frequency threshold is in a synchronized state, deduplication is performed on the data whose access frequency is less than the frequency threshold in the local storage node;

[0022] If the data whose access frequency is less than the frequency threshold is in an unsynchronized state, the data whose access frequency is less than the frequency threshold is synchronized to the cloud storage node, and after successful synchronization to the cloud storage node, the data whose access frequency is less than the frequency threshold is deduplicated in the local storage node.

[0023] Optionally, determining the access frequency of the data in the local storage node includes: acquiring a preset time threshold, and determining the access frequency of the data in the local storage node within the time threshold.

[0024] Optionally, during the application-level deduplication, deduplication of non-local data in the same data accessed by different applications includes:

[0025] During application-level deduplication, local data in the same data accessed by different applications is determined based on the access conditions of the same data by different applications; different applications correspond to different local data;

[0026] Deduplication of non-local data in the same data accessed by different applications.

[0027] Optionally, it also includes:

[0028] For different applications, local data corresponding to the applications are retained.

[0029] Optionally, the different applications include applications using different data access protocols, and during the application-level deduplication, before deduplication of non-local data in the same data accessed by different applications, the method further includes:

[0030] Set a unified metadata format for different data access protocols.

[0031] Optionally, when deduplicating at the data block level, deduplicating the data blocks in the cloud storage node includes:

[0032] During data block-level deduplication, determining the data heat of the data blocks in the cloud storage node;

[0033] Deduplication is performed on data blocks in the cloud storage node whose data heat is less than a low heat threshold.

[0034] Optionally, it also includes:

[0035] For data blocks in the cloud storage node whose data heat is greater than a high heat threshold, a copy data block is created, and the copy database is stored in other cloud storage nodes;

[0036] The high heat threshold is greater than the low heat threshold.

[0037] Optionally, it also includes:

[0038] For data blocks in the cloud storage node whose data heat is greater than or equal to a low heat threshold and less than or equal to a high heat threshold, deduplication and copy creation are stopped.

[0039] Optionally, during the byte-level deduplication, deduplication of byte segments in the data blocks of the storage system includes:

[0040] During byte-level deduplication, the data blocks of the storage system are divided into byte segments, and corresponding fingerprint information is generated for each byte segment;

[0041] Determine byte segments with the same fingerprint information in the data block;

[0042] Deduplication is performed on byte segments with the same fingerprint information.

[0043] Optionally, it also includes:

[0044] Obtain deduplication logs returned by the deduplication operations of the multiple granularities.

[0045] A data processing device, wherein a storage system is provided with a cloud storage node and a local storage node, and the device comprises:

[0046] A multi-granularity deduplication module, configured to perform deduplication operations of multiple granularities on the storage system in response to a data deduplication request;

[0047] A system-level deduplication module, used for deduplicating data in the local storage node during system-level deduplication;

[0048] Application-level deduplication module, used to deduplicate non-local data in the same data accessed by different applications during application-level deduplication;

[0049] A data block level deduplication module, used for deduplicating data blocks in the cloud storage node during data block level deduplication;

[0050] The byte-level deduplication module is used to deduplicate byte segments in the data blocks of the storage system during byte-level deduplication.

[0051] The embodiments of the present invention have the following advantages:

[0052] In an embodiment of the present invention, a storage system is provided with a cloud storage node and a local storage node. By responding to a data deduplication request, deduplication operations of various granularities are performed on the storage system. In system-level deduplication, data in the local storage node is deduplicated. In application-level deduplication, non-local data in the same data accessed by different applications is deduplicated. In data block-level deduplication, data blocks in the cloud storage node are deduplicated. In byte-level deduplication, byte fragments in data blocks of the storage system are deduplicated. This achieves data deduplication of the storage system at various granularities and improves the data deduplication rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 is a flowchart of steps of a data processing method provided by some embodiments of the present invention;

[0055] Figure 2 is a schematic diagram of a system architecture provided by some embodiments of the present invention;

[0056] Figure 3 is a schematic diagram of a deduplication operation with multiple granularities provided by some embodiments of the present invention;

[0057] Figure 4 is a schematic diagram of system-level deduplication provided by some embodiments of the present invention;

[0058] Figure 5 is a schematic diagram of application-level deduplication provided by some embodiments of the present invention;

[0059] Figure 6 is a schematic diagram of data block-level deduplication provided by some embodiments of the present invention;

[0060] Figure 7 is a flowchart of another data processing method provided by some embodiments of the present invention;

[0061] Figure 8 is a structural block diagram of a data processing device provided by some embodiments of the present invention;

[0062] Fig. 9 is a structural block diagram of an electronic device provided by some embodiments of the present invention;

[0063] Fig.10 is a structural block diagram of a storage medium provided by some embodiments of the present invention;

[0064] Fig.11 It is a structural block diagram of a program product provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] With the explosive growth of data scale, data of different protocol types is also increasing. Deduplication technology plays an important role in improving the performance of storage systems, reducing operating costs and improving data management efficiency, which is mainly reflected in the following aspects:

[0067] Save storage space: By deleting duplicate data copies, storage capacity requirements are reduced, thereby reducing storage costs and resource waste; Improve storage efficiency: Reduce data redundancy so that the storage system can use storage resources more efficiently, improve data access speed and overall performance; Reduce backup and recovery time: During the backup and recovery process, removing duplicate data can shorten the backup time and reduce the time required for data recovery, thereby improving data protection efficiency; Reduce network bandwidth usage: During data transmission (such as backup or synchronization), reducing duplicate data can reduce network bandwidth consumption and improve transmission speed and efficiency.

[0068] In the deduplication method of the relevant data block granularity, the full SHA-1 or SHA-2 fingerprint calculation is required, and the fingerprint information must be completely matched, resulting in a low data reduction rate between systems and different protocol applications, and a large CPU computing overhead, making it difficult to meet the storage system's demand for space utilization. In the cloud distributed storage scenario, the low data deduplication rate leads to high data redundancy, and the transmission between nodes causes a high network resource usage problem; in addition, a large amount of redundant data stored on the disk will also reduce the disk life, affecting data security, as well as system reliability and availability.

[0069] Although the byte-level deduplication method can effectively save storage space, it also has some disadvantages, as follows:

[0070] High computational overhead: Due to the need for byte-by-byte comparison and hash calculation, byte-level deduplication consumes more computing resources and time, which may lead to performance degradation, especially when processing large-scale data. High memory consumption: The need to store and manage a large number of hash values ​​and metadata may lead to high memory consumption, especially when the amount of data is large. High complexity: Implementing byte-level deduplication requires processing complex data structures and algorithms, which may increase the complexity of system implementation and maintenance. Low deduplication efficiency: For data with a high change rate or frequent updates, byte-level deduplication may result in lower deduplication efficiency because the system still needs to process a large number of hash values ​​and comparison operations even if there are only a few changes in the data. Security risks: If there are vulnerabilities in the hash function, it may lead to security risks, such as hash collision attacks, which cause different byte sequences to produce the same hash value, affecting the deduplication effect and data integrity.

[0071] In the embodiment of the present invention, in the storage system, according to the access characteristics of data, a multi-granularity deduplication mechanism is designed from different data access levels to achieve cross-system and cross-application global adaptive deduplication of duplicate data, as follows:

[0072] In system-level deduplication, by sensing the usage frequency of objects or files, data that has not been used for a long time is transferred or synchronized to a system with low storage cost for storage, and the data copy of the local system is deleted to save local storage cost and space. In application-level deduplication, a protocol fusion mechanism is designed so that multiple applications can access the same data with different protocols, reducing data redundancy and improving system availability. At the data block (chunk) granularity, different fingerprint information of each data block is calculated and compared to achieve data deduplication at the data block granularity; then, a byte-level deduplication method is used to further compress the stored data.

[0073] In addition, in response to the problems of data access consistency and data access concurrency performance, we have designed a cross-protocol deduplication solution and a data block replica adaptive increase and decrease mechanism. That is, in response to the situation where different protocol applications access the same data, we design a cross-protocol deduplication mechanism based on the locality of data modification operations by different applications. By deduplicating non-local data, we can improve the system storage efficiency and data access flexibility. In response to the problem of data access concurrency performance, we design a mechanism for adaptively adjusting the number of data block replicas based on the popularity of the data block to improve the system's response efficiency and performance.

[0074] Moreover, the deduplication operations at each level are performed serially and in parallel. Specifically, when deduplication is performed in parallel at each level, it is performed according to the established fingerprint matching strategy. When a certain level senses that the deduplication operation at the previous level has an impact on the deduplication object at the current level, the current level needs to wait until the previous level completes the operation on the object or file before performing the deduplication operation, which can improve the speed and efficiency of deduplication.

[0075] In the embodiments of the present invention, by constructing deduplication mechanisms of different granularities, duplicate data can be deleted from the storage data access level, which can improve the storage efficiency of distributed cloud storage nodes in many aspects, thereby increasing the availability and data reliability of the system. In addition, for applications of different protocols, a cross-protocol deduplication solution is designed to reduce the storage space occupied by duplicate data and metadata, which can further improve the storage efficiency of data; for the system concurrency performance problem caused by deduplication, a method for adaptively increasing and decreasing the number of copies is designed, which can ensure the storage space efficiency while improving the data access performance of the system.

[0076] Compared with the byte-level deduplication method of the related art, the embodiment of the present invention can enhance the deduplication effect, improve the data reduction rate, improve the efficiency of the deduplication operation, and improve the overall space efficiency of the storage system and the life and reliability of the storage device by deduplicating data at different data granularity levels.

[0077] First, deduplication is performed between systems. A large amount of duplicate data can be filtered out from the file or object level and deleted, avoiding repeated operations of data block-level and byte-level deduplication, thereby improving the overall deduplication efficiency. Then, deduplication is performed between applications of different protocols. The protocol fusion method is adopted to enable different protocol applications to read and write the same data and metadata, which can improve the flexibility and availability of the storage system. In addition, a multi-protocol fusion deduplication technology is designed between different applications, and the locality of data block modifications is identified. Different data versions can be provided for different applications, and further data reduction at the block level can be achieved. Finally, combined with the byte-level deduplication method in the relevant technology, data compression at the data block level can be achieved to ensure the final data reduction rate of the stored data.

[0078] In an embodiment of the present invention, through data deduplication operations at multiple granularity levels, combined with the designed heat-aware replica adaptive replication and deduplication mechanism, and the deduplication technology based on protocol fusion and data modification locality, the data deduplication effect can be improved while enhancing the system performance.

[0079] The present invention is further described below in conjunction with the accompanying drawings:

[0080] Reference Figure 1 , showing a step flow chart of a data processing method provided by some embodiments of the present invention, the storage system is provided with a cloud storage node and a local storage node.

[0081] In some embodiments of the present invention, the storage system includes any one of the following:

[0082] Distributed storage system, centralized storage system.

[0083] like Figure 2 In a distributed storage system, there are n cloud storage nodes, m local storage nodes, and 1 cloud computing server. The local storage node is connected to the cloud computing server and the cloud storage node through a network (such as Ethernet, Internet, local area network or wide area network, etc.).

[0084] Specifically, the following steps may be included:

[0085] Step 101: In response to a data deduplication request, perform deduplication operations of multiple granularities on the storage system.

[0086] In some examples, the deduplication operation is to delete duplicate data. The deduplication operation includes system-level deduplication, application-level deduplication, data block-level deduplication, byte-level deduplication, and of course, there may be other deduplication operations.

[0087] Among them, data deduplication technology automatically searches for duplicate data in the data set, retains only one copy of the same data, and uses a pointer to the copy to replace other duplicate data copies, thereby reducing storage space requirements. Multi-granularity deduplication refers to the use of deduplication technology at different granularity levels to achieve data reduction, such as object granularity, data block granularity, and byte-level granularity.

[0088] In actual applications, the multi-level deduplication solution is mainly divided into four parts: system-level deduplication, application-level deduplication, block-level deduplication and byte-level deduplication. Four granularities of deduplication are considered, namely, deduplication at the granularity of files and objects between local storage nodes and cloud storage nodes (i.e., system-level deduplication), deduplication of files and objects at the application level in cloud storage nodes (i.e., application-level deduplication), deduplication at the block level in cloud storage nodes (i.e., block-level deduplication), and deduplication at the byte level within each cloud storage node (i.e., byte-level deduplication).

[0089] In the storage system, multi-granularity deduplication is used to deduplicate stored data at the system level, application level, data block level, and byte level. In the multi-granularity data deduplication solution, serial and parallel deduplication implementation plans are designed for deduplication operations at each granularity.

[0090] In some embodiments of the present invention, the deduplication operations of the various granularities have a hierarchical relationship, and the hierarchical relationship between the deduplication operations of the various granularities is as follows from the upper layer to the lower layer:

[0091] System-level deduplication, application-level deduplication, data block-level deduplication, and byte-level deduplication.

[0092] In some embodiments of the present invention, the method further comprises:

[0093] When executing a deduplication operation at the current level, determine whether the data to be deleted corresponding to the deduplication operation at the current level is within the deduplication data range of the deduplication operation at the previous level; if the data to be deleted is within the deduplication data range of the deduplication operation at the previous level, stop the deduplication operation on the data to be deleted at the current level; if the data to be deleted is outside the deduplication data range of the deduplication operation at the previous level, perform a deduplication operation on the data to be deleted at the current level.

[0094] In the storage system, the system level, application level, data block level and byte level perform data deduplication according to the deduplication operation trigger conditions set by each level. Among them, the application level, data block level and byte level can sense the deduplication operation of the previous layer. When the deduplication operation of the previous layer is related to the content being deduplicated at this layer, the current layer stops deduplication of the data and continues to dedupe other data.

[0095] Specifically, after the deduplication request conditions are met, it can be scheduled to each level. In the application-level, data block-level, and byte-level deduplication operations, if the data to be deleted is not within the deduplication range of the previous level, the deduplication operation of the data is performed; if the data to be deleted is within the deduplication range of the previous level, the deduplication operation of the data is stopped and the deduplication of other data is continued.

[0096] like Figure 3If the data D1 to be deleted by application-level deduplication does not belong to the data D0 to be deleted by system-level deduplication, then D1 is deduplicated; if the data D2 to be deleted by data block-level deduplication does not belong to the data D1 to be deleted by application-level deduplication, then D2 is deduplicated; if the data D3 to be deleted by byte-block-level deduplication does not belong to the data D2 to be deleted by data block-level deduplication, then D3 is deduplicated.

[0097] Step 102, during system-level deduplication, deduplication is performed on the data in the local storage node.

[0098] Among them, system-level deduplication refers to the use of deduplication technology at the system level to achieve the deletion of duplicate objects or file data. For example, the local system and the cloud storage system use deduplication technology to delete data copies that have not been used for a long time in the local system, thereby optimizing the efficiency of local storage space.

[0099] In some embodiments of the present invention, deduplicating data in the local storage node during system-level deduplication includes: determining the access frequency of the data in the local storage node during system-level deduplication; and deduplicating data in the local storage node whose access frequency is less than a frequency threshold.

[0100] In some embodiments of the present invention, determining the access frequency of data in the local storage node includes: obtaining a preset time threshold, and determining the access frequency of data in the local storage node within the time threshold.

[0101] In some embodiments of the present invention, deduplication of data having an access frequency less than a frequency threshold is performed in the local storage node, including: determining a synchronization status of the data having an access frequency less than the frequency threshold; if the data having an access frequency less than the frequency threshold is in a synchronized state, deduplication of the data having an access frequency less than the frequency threshold is performed in the local storage node; if the data having an access frequency less than the frequency threshold is in an unsynchronized state, synchronization of the data having an access frequency less than the frequency threshold to the cloud storage node, and after successful synchronization to the cloud storage node, deduplication of the data having an access frequency less than the frequency threshold is performed in the local storage node.

[0102] In system-level deduplication, such as Figure 4 , set a time threshold for a local storage node and a frequency threshold According to the file metadata information, the access frequency h (also called the heat threshold) of the file within the time threshold is obtained. If the access frequency h is greater than or equal to the frequency threshold No processing is done for the time being; if the access frequency h is less than the threshold The cloud storage and local file information are compared to see if they are synchronized. If the data is in a synchronized state, the local copy of the relevant data is deleted; if the data is in an unsynchronized state, the data is first synchronized to the cloud storage node, and then the local copy is deleted.

[0103] In some examples, in addition to considering the access frequency, the time since the last access may also be considered. If the time since the last access is greater than a time threshold, deduplication may be performed.

[0104] In the embodiment of the present invention, deduplication is performed between systems, and a large amount of duplicate data can be screened out from the file or object level and deleted, thereby avoiding repeated operations of data block level and byte level deduplication, thereby improving the deduplication efficiency as a whole.

[0105] Step 103, during application-level deduplication, deduplication is performed on non-local data in the same data accessed by different applications.

[0106] In some embodiments of the present invention, during application-level deduplication, non-local data in the same data accessed by different applications is deduplicated, including: during application-level deduplication, local data in the same data accessed by different applications is determined based on access conditions of the same data by different applications; non-local data in the same data accessed by different applications is deduplicated.

[0107] Among them, different applications correspond to different local data.

[0108] In some embodiments of the present invention, it also includes:

[0109] For different applications, local data corresponding to the applications are retained.

[0110] In some embodiments of the present invention, the different applications include applications that use different data access protocols. During application-level deduplication, before deduplicating non-local data in the same data accessed by different applications, it also includes: setting a unified metadata format for different data access protocols.

[0111] In some examples, local data may be the unique access portion of the same data for each application, while non-local data is the access portion of the same data shared by all applications. For example, different applications are interested in similar content of a picture and have roughly the same editing scope, so non-local data exists.

[0112] In application-level deduplication, such as Figure 5First, a unified metadata format is designed for different data access protocols (such as S3 objects, file protocols, etc.) to achieve protocol fusion and cross-protocol access to a piece of data; according to the access locality characteristics of different applications, that is, the locality of reading and writing the same data, data blocks other than the locality characteristics (that is, non-local data) are deduplicated, and the metadata and locality data corresponding to the protocol are retained, thereby achieving application-level deduplication of duplicate data.

[0113] Among them, protocol fusion refers to the fusion of different data access protocols, that is, different protocols can access the same data and metadata. The main content is the design and unification of metadata formats.

[0114] In the embodiment of the present invention, duplicate data is deleted between applications of different protocols, and a protocol fusion method is adopted to enable different protocol applications to read and write the same data and metadata, which can improve the flexibility and availability of the storage system. In addition, a multi-protocol fusion deduplication technology between different applications is designed, and the locality of data block modification is identified, so that different data versions can be provided for different applications, and further data reduction at the block level can be achieved.

[0115] Step 104, when performing data block-level deduplication, deduplication is performed on the data blocks in the cloud storage node.

[0116] Among them, data block (Chunk) level deduplication: In storage systems, object or file data is often divided into data blocks of the same size, also known as chunks. This deduplication technology calculates and compares fingerprints of data blocks and deletes data blocks with the same fingerprints.

[0117] In some embodiments of the present invention, deduplication is performed on data blocks in the cloud storage node when deduplication is performed at the data block level, including: determining the data heat of the data blocks in the cloud storage node when deduplication is performed at the data block level; and deduplication is performed on data blocks in the cloud storage node whose data heat is less than a low heat threshold.

[0118] In some embodiments of the present invention, it also includes: for data blocks in the cloud storage node whose data heat is greater than a high heat threshold, creating a copy data block, and storing the copy data block in other cloud storage nodes; wherein the high heat threshold is greater than the low heat threshold.

[0119] In some embodiments of the present invention, the method further includes: stopping deduplication and replica creation for data blocks in the cloud storage node whose data heat is greater than or equal to a low heat threshold and less than or equal to a high heat threshold.

[0120] Data heat refers to the measurement of data access frequency, which is used to distinguish the activity of data in the storage system. According to data heat, data can be classified into hot data, warm data and cold data.

[0121] In block-level deduplication, Figure 6 , count the access heat of each data block in the cloud storage node, and set two heat thresholds: low heat threshold f l and high heat threshold f h .

[0122] For data blocks with a temperature lower than the low temperature threshold, deduplication is performed to reduce the storage space occupied by the data blocks; for data blocks with a temperature higher than the high temperature threshold, replication is used to send the replica data blocks to several other cloud storage nodes to improve the concurrent performance of data access. When the temperature of the data blocks drops, the temperature classification and deduplication operations are performed on them; for data blocks between the low temperature threshold and the high temperature threshold, deduplication and replication operations are not performed.

[0123] In the embodiment of the present invention, in order to solve the system concurrency performance problem caused by deduplication, the number of copies is designed to increase and decrease adaptively, which can ensure the storage space efficiency while improving the data access performance of the system.

[0124] Step 105, during byte-level deduplication, deduplication is performed on byte segments in the data blocks of the storage system.

[0125] Among them, byte-level deduplication is a data deduplication technology that detects and deletes duplicate byte data by dividing the data into blocks and calculating hash values, thereby saving storage space. This technology is used in backup and storage systems to improve storage efficiency and reduce storage costs.

[0126] In some embodiments of the present invention, when deduplicating at the byte level, deduplicating byte segments in the data blocks of the storage system includes: when deduplicating at the byte level, dividing the data blocks of the storage system into byte segments, and generating corresponding fingerprint information for each byte segment; determining byte segments with the same fingerprint information in the data blocks; and deduplicating the byte segments with the same fingerprint information.

[0127] In some embodiments of the present invention, the fingerprint information includes a hash value.

[0128] Fingerprint calculation refers to cryptographically secure hash signatures, which are a method for uniquely identifying data, such as SHA-1 or SHA-2, and are usually used in scenarios such as data deduplication, data integrity verification, and data similarity detection. Fingerprint calculation represents the original data content by converting the data into a shorter unique identifier (fingerprint), such as fingerprint calculation methods including hash functions and various encryption algorithms.

[0129] In byte-level deduplication, the distributed storage system uses a window filtering method to divide the data block into smaller byte segments, each of which generates a unique hash value. The system compares these hash values, identifies and deletes duplicate segments, and retains only one copy. This can effectively save storage space and improve storage efficiency while ensuring data integrity and consistency.

[0130] In some embodiments of the present invention, it also includes:

[0131] Obtain deduplication logs returned by the deduplication operations of the multiple granularities.

[0132] In an embodiment of the present invention, a storage system is provided with a cloud storage node and a local storage node. By responding to a data deduplication request, deduplication operations of various granularities are performed on the storage system. In system-level deduplication, data in the local storage node is deduplicated. In application-level deduplication, non-local data in the same data accessed by different applications is deduplicated. In data block-level deduplication, data blocks in the cloud storage node are deduplicated. In byte-level deduplication, byte fragments in data blocks of the storage system are deduplicated. This achieves data deduplication of the storage system at various granularities and improves the data deduplication rate.

[0133] First, through multi-granular storage data deduplication, the cloud distributed storage system can be deduplicated globally, which can achieve a higher deduplication rate; and through the layer-by-layer deduplication solution, the amount of deduplication calculation can be reduced, and the deduplication speed and efficiency can be improved. At the system level, deduplication can reduce storage costs by placing local cold data on cloud storage nodes with lower storage costs; at the application level, deduplication can be performed on application data between different protocols, improving system storage efficiency and access flexibility; at the data block level, deduplication can be performed by adaptively increasing or decreasing the number of data block copies, which can achieve efficient data reduction while meeting data access performance requirements; finally, combined with byte-level deduplication methods, system data can be further deduplicated to improve space utilization.

[0134] Reference Figure 7 , shows a flowchart of another data processing method provided by some embodiments of the present invention, which may specifically include the following steps:

[0135] Step 701: In response to a data deduplication request, perform deduplication operations of multiple granularities on the storage system.

[0136] Step 702: When executing the deduplication operation at the current level, determine whether the to-be-deleted data corresponding to the deduplication operation at the current level is within the deduplication data range of the deduplication operation at the previous level.

[0137] Step 703: If the data to be deleted is within the deduplication data range of the deduplication operation at the previous level, the deduplication operation on the data to be deleted at the current level is stopped.

[0138] Step 704: If the data to be deleted is outside the deduplication data range of the deduplication operation at the previous level, a deduplication operation is performed on the data to be deleted at the current level.

[0139] Step 705: During system-level deduplication, deduplication is performed on the data in the local storage node.

[0140] Step 706, during application-level deduplication, deduplication is performed on non-local data in the same data accessed by different applications.

[0141] Step 707: When deduplication is performed at the data block level, deduplication is performed on the data blocks in the cloud storage node.

[0142] Step 708: During byte-level deduplication, byte segments in the data blocks of the storage system are deduplicated.

[0143] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0144] Reference Figure 8 , showing a structural schematic diagram of a data processing device provided in some embodiments of the present invention, the storage system is provided with a cloud storage node and a local storage node.

[0145] Specifically, the following modules may be included:

[0146] A multi-granularity deduplication module 801, configured to perform deduplication operations of multiple granularities on the storage system in response to a data deduplication request;

[0147] A system-level deduplication module 802, configured to perform deduplication on data in the local storage node during system-level deduplication;

[0148] An application-level deduplication module 803 is used to deduplicate non-local data in the same data accessed by different applications during application-level deduplication;

[0149] A data block level deduplication module 804, configured to perform deduplication on data blocks in the cloud storage node during data block level deduplication;

[0150] The byte-level deduplication module 805 is used to deduplicate byte segments in the data blocks of the storage system during byte-level deduplication.

[0151] Optionally, the deduplication operations of the multiple granularities have a hierarchical relationship, and the device further includes:

[0152] A data range determination module, used to determine whether the to-be-deleted data corresponding to the deduplication operation at the current level is within the deduplication data range of the deduplication operation at the previous level when executing the deduplication operation at the current level;

[0153] A deduplication stopping module, configured to stop the deduplication operation on the data to be deleted at the current level if the data to be deleted is within the deduplication data range of the deduplication operation at the previous level;

[0154] The deduplication operation execution module is used to execute the deduplication operation on the data to be deleted at the current level if the data to be deleted is outside the deduplication data range of the deduplication operation at the previous level.

[0155] Optionally, the hierarchical relationship between the deduplication operations of the multiple granularities is as follows from the upper layer to the lower layer:

[0156] System-level deduplication, application-level deduplication, data block-level deduplication, and byte-level deduplication.

[0157] Optionally, during the system-level deduplication, deduplication of data in the local storage node includes:

[0158] During system-level deduplication, determining the access frequency of data in the local storage node;

[0159] In the local storage node, deduplication is performed on the data whose access frequency is less than a frequency threshold.

[0160] Optionally, in the local storage node, deduplicating the data whose access frequency is less than a frequency threshold includes:

[0161] Determining a synchronization state of the data whose access frequency is less than a frequency threshold;

[0162] If the data whose access frequency is less than the frequency threshold is in a synchronized state, deduplication is performed on the data whose access frequency is less than the frequency threshold in the local storage node;

[0163] If the data whose access frequency is less than the frequency threshold is in an unsynchronized state, the data whose access frequency is less than the frequency threshold is synchronized to the cloud storage node, and after successful synchronization to the cloud storage node, the data whose access frequency is less than the frequency threshold is deduplicated in the local storage node.

[0164] Optionally, determining the access frequency of the data in the local storage node includes: acquiring a preset time threshold, and determining the access frequency of the data in the local storage node within the time threshold.

[0165] Optionally, during the application-level deduplication, deduplication of non-local data in the same data accessed by different applications includes:

[0166] During application-level deduplication, local data in the same data accessed by different applications is determined based on the access conditions of the same data by different applications; different applications correspond to different local data;

[0167] Deduplication of non-local data in the same data accessed by different applications.

[0168] Optionally, it also includes:

[0169] The local data retaining module is used to retain the local data corresponding to different applications.

[0170] Optionally, the different applications include applications using different data access protocols, and during the application-level deduplication, before deduplication of non-local data in the same data accessed by different applications, the method further includes:

[0171] The unified format setting module is used to set a unified metadata format for different data access protocols.

[0172] Optionally, when deduplicating at the data block level, deduplicating the data blocks in the cloud storage node includes:

[0173] During data block-level deduplication, determining the data heat of the data blocks in the cloud storage node;

[0174] Deduplication is performed on data blocks in the cloud storage node whose data heat is less than a low heat threshold.

[0175] Optionally, it also includes:

[0176] A data synchronization module, for creating a replica data block for a data block in the cloud storage node whose data heat is greater than a high heat threshold, and storing the replica data block in other cloud storage nodes;

[0177] The high heat threshold is greater than the low heat threshold.

[0178] Optionally, it also includes:

[0179] The deduplication and replica creation stopping module is used to stop deduplication and replica creation for data blocks whose data heat in the cloud storage node is greater than or equal to a low heat threshold and less than or equal to a high heat threshold.

[0180] Optionally, during the byte-level deduplication, deduplication of byte segments in the data blocks of the storage system includes:

[0181] During byte-level deduplication, the data blocks of the storage system are divided into byte segments, and corresponding fingerprint information is generated for each byte segment;

[0182] Determine byte segments with the same fingerprint information in the data block;

[0183] Deduplication is performed on byte segments with the same fingerprint information.

[0184] Optionally, it also includes:

[0185] The deduplication log acquisition module is used to obtain the deduplication logs returned by the deduplication operations of the multiple granularities.

[0186] Reference Fig. 9 , shows an electronic device provided by some embodiments of the present invention, including a processor 901, a memory 902, and a computer program stored in the memory 902 and capable of running on the processor 901, and the above method is implemented when the computer program is executed by the processor.

[0187] Reference Fig.10 , shows a computer-readable storage medium 1000 provided in some embodiments of the present invention, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0188] Reference Fig.11 , shows a computer program product 1100 provided by some embodiments of the present invention, including a computer program, and the computer program implements the above method when executed by a processor.

[0189] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0191] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0192] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0193] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0197] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0198] The data processing method and device provided above are introduced in detail. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: The storage system is provided with a cloud storage node and a local storage node, and the method comprises: In response to a data deduplication request, performing deduplication operations of multiple granularities on the storage system; During system-level deduplication, deduplication is performed on the data in the local storage node; During application-level deduplication, deduplication is performed on non-local data in the same data accessed by different applications; In case of data block level deduplication, deduplication is performed on the data blocks in the cloud storage node; In byte-level deduplication, byte segments in data blocks of the storage system are deduplicated.

2. The method according to claim 1, characterized in that The deduplication operations of the multiple granularities have a hierarchical relationship, and the method further includes: When executing a deduplication operation at the current level, determining whether the to-be-deleted data corresponding to the deduplication operation at the current level is within the deduplication data range of the deduplication operation at the previous level; If the data to be deleted is within the deduplication data range of the deduplication operation at the previous level, then the deduplication operation on the data to be deleted at the current level is stopped; If the data to be deleted is outside the deduplication data range of the deduplication operation at the previous level, a deduplication operation is performed on the data to be deleted at the current level.

3. The method according to claim 2, characterized in that The hierarchical relationship between the deduplication operations of the various granularities is as follows from the upper layer to the lower layer: System-level deduplication, application-level deduplication, data block-level deduplication, and byte-level deduplication.

4. The method according to any one of claims 1 to 3, characterized in that: When deduplicating at the system level, deduplicating data in the local storage node includes: During system-level deduplication, determining the access frequency of data in the local storage node; In the local storage node, deduplication is performed on the data whose access frequency is less than a frequency threshold.

5. The method according to claim 4, characterized in that Deduplication of the data having an access frequency less than a frequency threshold in the local storage node includes: Determining a synchronization state of the data whose access frequency is less than a frequency threshold; If the data whose access frequency is less than the frequency threshold is in a synchronized state, deduplication is performed on the data whose access frequency is less than the frequency threshold in the local storage node; If the data whose access frequency is less than the frequency threshold is in an unsynchronized state, the data whose access frequency is less than the frequency threshold is synchronized to the cloud storage node, and after successful synchronization to the cloud storage node, the data whose access frequency is less than the frequency threshold is deduplicated in the local storage node.

6. The method according to claim 4, characterized in that The determining the access frequency of the data in the local storage node includes: obtaining a preset time threshold, and determining the access frequency of the data in the local storage node within the time threshold.

7. The method according to any one of claims 1 to 3, characterized in that: During application-level deduplication, deduplication is performed on non-local data in the same data accessed by different applications, including: During application-level deduplication, local data in the same data accessed by different applications is determined based on the access conditions of the same data by different applications; different applications correspond to different local data; Deduplication of non-local data in the same data accessed by different applications.

8. The method according to claim 7, characterized in that Also includes: For different applications, local data corresponding to the applications are retained.

9. The method according to claim 7, characterized in that: The different applications include applications using different data access protocols. When deduplicating at the application level, before deduplicating non-local data in the same data accessed by different applications, the method further includes: Set a unified metadata format for different data access protocols.

10. The method according to any one of claims 1 to 3, characterized in that: When deduplicating at the data block level, deduplicating the data blocks in the cloud storage node includes: During data block-level deduplication, determining the data heat of the data blocks in the cloud storage node; Deduplication is performed on data blocks in the cloud storage node whose data heat is less than a low heat threshold.

11. The method according to claim 10, characterized in that Also includes: For a data block in the cloud storage node whose data heat is greater than a high heat threshold, a copy data block is created, and the copy data block is stored in another cloud storage node; The high heat threshold is greater than the low heat threshold.

12. The method according to claim 11, characterized in that Also includes: For data blocks in the cloud storage node whose data heat is greater than or equal to a low heat threshold and less than or equal to a high heat threshold, deduplication and copy creation are stopped.

13. The method according to any one of claims 1 to 3, characterized in that: During byte-level deduplication, deduplication of byte segments in data blocks of the storage system includes: During byte-level deduplication, the data blocks of the storage system are divided into byte segments, and corresponding fingerprint information is generated for each byte segment; Determine byte segments with the same fingerprint information in the data block; Deduplication is performed on byte segments with the same fingerprint information.

14. The method according to any one of claims 1 to 3, characterized in that: Also includes: Obtain deduplication logs returned by the deduplication operations of the multiple granularities.

15. A data processing device, characterized in that: The storage system is provided with a cloud storage node and a local storage node, and the device comprises: A multi-granularity deduplication module, configured to perform deduplication operations of multiple granularities on the storage system in response to a data deduplication request; A system-level deduplication module, used for deduplicating data in the local storage node during system-level deduplication; Application-level deduplication module, used to deduplicate non-local data in the same data accessed by different applications during application-level deduplication; A data block level deduplication module, used for deduplicating data blocks in the cloud storage node during data block level deduplication; The byte-level deduplication module is used to deduplicate byte segments in the data blocks of the storage system during byte-level deduplication.

Images