Repeated data processing method and device, storage medium and computer equipment

Abstract

The invention provides a duplicated data processing method and device, a storage medium and computer equipment, and the method comprises the steps: obtaining a data block of a target object uploaded by a client at a time, the data block being a part of object data of the target object; an original identifier of initial deduplication metadata of the data block is calculated, and the initial deduplication metadata is used for representing whether the data block exists in the data pool or not; under the condition that it is determined that initial de-duplication metadata exists in the index pool according to the original identifier, it is determined that the data blocks are duplicated data, the data blocks are not stored in the data pool any more, and attribute information of the data blocks is updated; the index pool is used for storing initial de-duplication metadata, and the data pool is used for storing object data of the target object. The uploading efficiency of the data object is improved, repeated data blocks are prevented from being written into the data pool, the storage space of the data pool is saved, and then the utilization rate of the storage space in the storage system is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a method, device, storage medium and computer equipment for processing duplicate data. Background Art

[0002] In the field of data storage, object storage is particularly important due to its flexibility and scalability. Object storage provides higher scalability and easier management by managing data as objects rather than traditional files or blocks. This storage method allows large amounts of data to be stored in the cloud in the form of simple objects with metadata, so that data can be flexibly expanded according to demand. However, a considerable portion of the data stored in the object storage system is duplicate data, so a deduplication module is needed to deduplicate the uploaded objects.

[0003] Currently, the deletion of duplicate data is done after the entire object data is uploaded. It is determined whether the object data exists in the data pool. If the object data already exists in the data pool, the object data is deleted. This leads to a relatively low efficiency in deleting duplicate object data. Summary of the invention

[0004] In view of this, the present disclosure provides a method, apparatus, storage medium and computer equipment for processing duplicate data, which can improve the utilization rate of storage space in a storage system.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing duplicate data is provided, the method comprising:

[0006] Acquire a data block of a target object uploaded by a client in a single time, wherein the data block is a part of the object data of the target object;

[0007] Calculating an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in the data pool;

[0008] When it is determined based on the original identifier that the initial deduplication metadata exists in the index pool, the data block is determined to be duplicate data, the data block is no longer stored in the data pool, and the attribute information of the data block is updated; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

[0009] In one embodiment, the calculating the original identifier corresponding to the data block includes:

[0010] Calculate the digest value corresponding to the data block according to the digest algorithm;

[0011] Calculating a storage node of initial deduplication metadata corresponding to the data block according to the summary value;

[0012] Determine the name of the data pool according to the name of the storage bucket corresponding to the data block;

[0013] The original identifier is calculated according to the name of the data pool, the storage node and the digest value.

[0014] In one embodiment, the method further comprises:

[0015] In the case where the data block is a starting data block of the target object, generating an information recording table of the target object and segmentation information of the data block, wherein the information recording table is used to record the segmentation information of all data blocks of the target object, and the segmentation information is used to indicate that the data block is stored in the data pool;

[0016] The segmentation information of the data block is written into the information recording table.

[0017] In one embodiment, the method further comprises:

[0018] In a case where the data block is a starting data block of the target object, determining a storage position code corresponding to a storage position of the data block in the data pool as a starting position code;

[0019] In the case that the data block is a non-starting data block of the target object, the storage position of the current data block is determined according to the storage position of the previous data block in the data pool.

[0020] In one embodiment, updating the attribute information of the data block includes:

[0021] Generate new deduplication metadata according to the name of the target object, the name of the storage bucket corresponding to the data block, and the initial deduplication metadata;

[0022] Generate a specific identifier corresponding to the new deduplication metadata according to the original identifier, wherein the number of the specific identifiers is used to represent the number of deduplication metadata in the index pool;

[0023] The new deduplication metadata and the specific identifier are stored in the index pool in correspondence, wherein the number of the deduplication metadata in the index pool is used to indicate the number of times the data block is referenced.

[0024] In one embodiment, after calculating the original identifier of the initial deduplication metadata corresponding to the data block, the method further includes:

[0025] When it is determined according to the original identifier that there is no initial deduplication metadata in the index pool, storing the data block in the data pool;

[0026] Generating initial deduplication metadata corresponding to the data block;

[0027] generating an original identifier of the data block according to the initial deduplication metadata;

[0028] Generating a specific identifier corresponding to the initial deduplication metadata according to the original identifier;

[0029] The original identifier, the initial deduplication metadata and the specific identifier are stored in the index pool in correspondence.

[0030] In one embodiment, the method further comprises:

[0031] When all data blocks of the target object are stored, generating object metadata according to the target object;

[0032] The object metadata is stored in the index pool.

[0033] According to a second aspect of an embodiment of the present disclosure, a device for processing duplicate data is provided, the device comprising:

[0034] An acquiring unit, configured to acquire a data block of a target object uploaded by a client in a single time, wherein the data block is a part of the object data of the target object;

[0035] a calculation unit, configured to calculate an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in the data pool;

[0036] The first determination unit is used to determine that the data block is duplicate data, no longer store the data block in the data pool, and update the attribute information of the data block when it is determined that the initial deduplication metadata exists in the index pool based on the original identifier; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

[0037] In one embodiment, the computing unit is used to:

[0038] Calculate the digest value corresponding to the data block according to the digest algorithm;

[0039] Calculating a storage node of initial deduplication metadata corresponding to the data block according to the summary value;

[0040] Determine the name of the data pool according to the name of the storage bucket corresponding to the data block;

[0041] The original identifier is calculated according to the name of the data pool, the storage node and the digest value.

[0042] In one embodiment, the apparatus further comprises:

[0043] A first generating unit is configured to generate, when the data block is a starting data block of the target object, an information recording table of the target object and segmentation information of the data block, wherein the information recording table is used to record segmentation information of all data blocks of the target object, and the segmentation information is used to indicate that the data block is stored in the data pool;

[0044] The writing unit is used to write the segmentation information of the data block into the information recording table.

[0045] In one embodiment, the apparatus further comprises:

[0046] A second determining unit is used to determine, when the data block is a starting data block of the target object, that a storage position code corresponding to a storage position of the data block in the data pool is a starting position code;

[0047] The third determining unit is configured to determine the storage position of the current data block according to the storage position of the previous data block in the data pool when the data block is a non-starting data block of the target object.

[0048] In one embodiment, the first determining unit is configured to:

[0049] Generate new deduplication metadata according to the name of the target object, the name of the storage bucket corresponding to the data block, and the initial deduplication metadata;

[0050] Generate a specific identifier corresponding to the new deduplication metadata according to the original identifier, wherein the number of the specific identifiers is used to represent the number of deduplication metadata in the index pool;

[0051] The new deduplication metadata and the specific identifier are stored in the index pool in correspondence, wherein the number of the deduplication metadata in the index pool is used to indicate the number of times the data block is referenced.

[0052] In one embodiment, the apparatus further comprises:

[0053] a first storage unit, configured to store the data block in the data pool when it is determined according to the original identifier that no initial deduplication metadata exists in the index pool;

[0054] A second generating unit, configured to generate initial deduplication metadata corresponding to the data block;

[0055] A third generating unit, configured to generate an original identifier of the data block according to the initial deduplication metadata;

[0056] a fourth generating unit, configured to generate a specific identifier corresponding to the initial deduplication metadata according to the original identifier;

[0057] The second storage unit is used to store the original identifier, the initial deduplication metadata and the specific identifier in the index pool in correspondence.

[0058] In one embodiment, the apparatus further comprises:

[0059] A fifth generating unit, configured to generate object metadata according to the target object when all data blocks of the target object are stored;

[0060] The third storage unit is used to store the object metadata in the index pool.

[0061] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0062] According to a fourth aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0063] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0064] By splitting the target object into data blocks for storage and deduplicating the data when the same data blocks exist in the data pool, that is, not uploading duplicate data blocks to the data pool, the uploading efficiency of the data object is improved, and at the same time, writing duplicate data blocks to the data pool is avoided, saving the storage space of the data pool, thereby improving the utilization of the storage space in the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of a method for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0066] Figure 2 is a flow chart of a method for calculating an original identifier shown in an exemplary embodiment of the present disclosure;

[0067] Figure 3 is a flow chart of a method for updating data block attribute information shown in an exemplary embodiment of the present disclosure;

[0068] Figure 4 is a flow chart of a method for processing non-duplicate data shown in an exemplary embodiment of the present disclosure;

[0069] Figure 5 is a flow chart of another method for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0070] Figure 6 is a block diagram of a device for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0071] Figure 7 It is a hardware structure diagram of a computer device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0073] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0074] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0075] Object data is stored in an object storage system in a flat structure. Each object includes data, metadata, and a unique identifier. Object storage systems do not rely on traditional directory structures, which enables them to process large-scale unstructured data more efficiently and provide higher scalability and flexibility. However, some of the data stored in object storage systems is duplicate data, so these duplicate data need to be processed so that only one copy of the object data is stored in the storage system to save storage system space.

[0076] At present, when deleting duplicate data in an object storage system, it is necessary to first upload the object data completely to the storage system, then use a digest algorithm to calculate the digest value of the object data, and determine whether object data with the same digest value already exists in the storage system based on the digest value. If so, the object data is determined to be duplicate data, and then the object data is deleted from the storage system. The current method for deleting duplicate data requires that the object data be uploaded completely to the storage system before it can be determined whether the object data already exists in the storage system. If so, it is deleted. Therefore, the target of deleting duplicate data is the entire object data, and the processing granularity is coarse, so the storage space reduction effect achieved is limited. In addition, the repeated operation of first writing and then deleting the object data will also lead to the problem of low efficiency in storing data in the storage system.

[0077] Therefore, the present disclosure provides a method for processing duplicate data, which is applied to an object storage system. The flowchart of the method is shown in Figure 1 , Figure 1 This is a flowchart of a method for processing duplicate data shown in an exemplary embodiment of the present disclosure, which may specifically include the following steps:

[0078] S101, obtaining a data block of a target object uploaded by a client in a single time, wherein the data block is a part of object data of the target object.

[0079] In some embodiments, the client uploads object data to the object storage system via HTTP (Hypertext Transfer Protocol). When the client transmits the object data, the name of the storage bucket corresponding to the target object has been determined. The client can upload multiple times so that the system can obtain the data of the target object, and the data information uploaded each time is the data block of the target object mentioned above. The multiple data blocks uploaded by the client multiple times can be combined into the object data of the target object.

[0080] S102: Calculate an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in a data pool.

[0081] In some embodiments, the original identifier is used to search for deduplication metadata in the index pool. The index pool is used to store deduplication metadata. If there is deduplication metadata corresponding to a data block in the index pool, it means that the same data block already exists in the data pool used to store the data block, and the data in the data block is duplicate data. In order to quickly determine whether there is deduplication metadata corresponding to a data block in the index pool, it is necessary to calculate the original identifier of the initial deduplication metadata of the data block. In one embodiment, the summary value of the data block can be calculated by a summary algorithm, and then the index pool and storage node where the initial deduplication metadata of the data block is located can be determined based on the summary value, and the name of the data pool can be determined based on the name of the storage bucket corresponding to the data block. Finally, the storage node, summary value and name of the data pool can be combined to calculate the original identifier. The further calculation process will be described in detail later.

[0082] S103, when it is determined based on the original identifier that the initial deduplication metadata exists in the index pool, it is determined that the data block is duplicate data, the data block is no longer stored in the data pool, and the attribute information of the data block is updated; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

[0083] In one embodiment, after searching the index pool for the initial deduplication metadata according to the original identifier, if the initial deduplication metadata exists in the index pool, it can be determined that the data block already exists in the storage system and is duplicate data, so the storage system no longer stores the data block in the data pool, and only needs to update the attribute information of the data block. The attribute information includes segmentation information and deduplication metadata for indicating the storage of the data block in the storage system.

[0084] By splitting the target object into data blocks for storage and deduplicating the data when the same data blocks exist in the data pool, that is, not uploading duplicate data blocks to the data pool, the uploading efficiency of the data object is improved, and at the same time, writing duplicate data blocks to the data pool is avoided, saving the storage space of the data pool, thereby improving the utilization of the storage space in the storage system.

[0085] The following embodiment will illustrate the process of calculating the original identifier corresponding to the data block, please refer to Figure 2 As shown in the flow chart, Figure 2 FIG. 1 is a flow chart of a method for calculating an original identifier according to an exemplary embodiment of the present disclosure, which may specifically include the following steps:

[0086] S201, calculating a digest value corresponding to the data block according to a digest algorithm.

[0087] S202: Calculate a storage node of initial deduplication metadata corresponding to the data block according to the summary value.

[0088] S203: Determine the name of the data pool according to the name of the storage bucket corresponding to the data block.

[0089] S204: Calculate the original identifier according to the name of the data pool, the storage node and the summary value.

[0090] In some embodiments, the MD5 value corresponding to the data block (i.e., the above-mentioned digest value) can be calculated by MD5 (Message-Digest Algorithm 5). Then, based on the correspondence between the digest value and the index pool, the index pool and storage node where the deduplication metadata of the data block are located can be determined. Furthermore, in order to improve the security of the data, the digest value can be encrypted, and the index pool and storage node where the deduplication metadata are located can be verified based on the encryption result. When the client uploads the data block to the storage system, the name of the storage bucket corresponding to the data block has been determined. Furthermore, based on the correspondence between the storage bucket and the data pool, the data pool where the storage bucket is located can be determined, thereby determining the name of the data pool to be used to store the data block. Finally, based on the name of the data pool, the storage node and the digest value, the original identifier can be calculated. For the specific calculation method, please refer to the method shown in formula (1):

[0091] Original identifier = vnode value + MD5 value + data_prefix (1)

[0092] The vnode value indicates the storage node, the MD5 value indicates the digest value, and data_prefix indicates the name of the data pool.

[0093] At this point, it can be determined that the deduplication metadata includes at least the data pool identifier used to store the data block, the name of the storage bucket, the name of the target object, the file name corresponding to the data block, the summary value, the storage location of the file corresponding to the data block in the data pool, and the size of the data block.

[0094] By calculating the original identifier of the deduplication metadata, it is possible to quickly determine whether the deduplication metadata exists in the index pool according to the original identifier, thereby improving the efficiency of searching for the deduplication metadata.

[0095] In order to enable the storage system to accurately store data blocks according to the attribute information of the data blocks during the storage process, it is necessary to generate a segmentation information to indicate the storage process of the data blocks. At the same time, in order to quickly understand the attribute information of each data block in the target object after the storage of the target object is completed, it is necessary to record the attribute information of each data block. Therefore, it is necessary to initialize the storage process of the target object. The specific method is as follows:

[0096] In the case where the data block is the starting data block of the target object, an information recording table of the target object and segmentation information of the data block are generated, wherein the information recording table is used to record the segmentation information of all data blocks of the target object, and the segmentation information is used to indicate that the data block is stored in the data pool; the segmentation information of the data block is written into the information recording table.

[0097] In some embodiments, if the data block is the first data block of the target object, i.e., the starting data block, then it is necessary to generate the segmentation information and information record table of the data block according to the name of the target object, the file name corresponding to the data block, the segmentation number of the data block (starting from 1), and the size of the data block. Specifically, the information record table can be an array for storing one or more segmentation information. Then, the segmentation information of the starting data block is written into the information record table, so that the attribute information of the initial data block can be recorded.

[0098] After the storage system has processed the starting data block, it can update the segment information according to the attribute information of the next data block, that is, according to the name of the target object, the file name corresponding to the data block, the segment number of the data block, and the size of the data block, determine the attribute information of the next data block, where the segment number starts from 1 and increases, and the segment number is the data block number. For example, the segment number of the starting data block of the target object is 1, and the segment number of the second data block is 2.

[0099] Since the embodiments of the present disclosure implement the storage of the target object by means of data block storage, and ultimately the data blocks are spliced ​​into a complete target object, the data blocks are stored in sequence, and the storage position of the next data block can be determined according to the storage position and size of the previous data block. In order to accurately determine the storage position of each data block, the storage position of the data block in the data pool needs to be processed, and the specific processing method is as follows:

[0100] In the case where the data block is the starting data block of the target object, the storage position code corresponding to the storage position of the data block in the data pool is determined as the starting position code; in the case where the data block is a non-starting data block of the target object, the storage position of the current data block is determined based on the storage position of the previous data block in the data pool.

[0101] In some embodiments, when the data block is the starting data block of the target object, a variable offset=0 can be declared, indicating that the storage position code corresponding to the storage position of the starting data block in the data pool is the starting position code. If the starting data block is not duplicate data and is successfully written into the data pool, the storage position of the next data block needs to be immediately after the starting data block. For example, the storage position code of the starting data block is offset=0, and the size of the starting data block is 2KB. After the starting data block is stored in the data pool, the position code of the second data block can be offset=2048. Obviously, in the above embodiment, the storage position code is an exemplary description in bytes, and the content of the specific storage position code can be changed according to the actual amount of data stored.

[0102] In some embodiments, if the data block is duplicate data that already exists in the data pool, the data block does not need to be stored in the data pool again, but the attribute information of the data block needs to be updated to facilitate accurate storage of subsequent data blocks. Figure 3 The flowchart shown may specifically include the following steps:

[0103] S301, generating new deduplication metadata according to the name of the target object, the name of the storage bucket corresponding to the data block, and the initial deduplication metadata.

[0104] S302: Generate a specific identifier corresponding to the new deduplication metadata according to the original identifier, wherein the number of the specific identifiers is used to represent the number of deduplication metadata in the index pool.

[0105] S303: Store the new deduplication metadata and the specific identifier in the index pool in correspondence, wherein the number of deduplication metadata in the index pool is used to indicate the number of times the data block is referenced.

[0106] In some embodiments, a new deduplication metadata is generated based on the name of the target object, the name of the storage bucket corresponding to the data block, the summary value in the initial deduplication metadata, the storage location of the file corresponding to the data block in the data pool, and other information. The number of deduplication metadata corresponding to a data block in the index pool is the number of times the data block is referenced by the object data. For example, if a data block exists in the first object, the second object, and the third object at the same time, then there will be three deduplication metadata corresponding to the data block in the index pool. The number of times the data block is referenced is determined based on the number of deduplication metadata, so that when the first object becomes ineffective and all the data blocks of the first object need to be deleted in the data pool, the data block is not deleted due to the existence of multiple deduplication metadata, so as to avoid data loss of the second object and the third object. At the same time, because the deduplication metadata exists in the index pool, when deleting all the data blocks of a certain storage object, it is impossible to directly and quickly determine the number of deduplication metadata in the index pool. Therefore, it is necessary to set a specific identifier for quickly retrieving and determining the number of deduplication metadata in the index pool. For the generation of feature representation, reference can be made to the method shown in formula (2):

[0107] Specific ID = original ID + # + requestld (2)

[0108] Here, requestld represents a random character string.

[0109] A random string is concatenated after the original identifier to generate a new specific identifier for deduplication metadata. A specific identifier indicates that there is a deduplication metadata in the index pool. After the specific identifier is generated, the specific identifier and the new deduplication metadata can be stored in the index pool together for reference by other data blocks in the future.

[0110] If it is determined according to the original identifier that there is no initial deduplication metadata in the index pool, that is, the data information of the data block does not exist in the data pool, then the data block needs to be stored in the data pool and the attribute information of the data block needs to be updated. For this process, see Figure 4 The flowchart shown, Figure 4 This is a flowchart of a method for processing non-duplicate data shown in an exemplary embodiment of the present disclosure, which may specifically include the following steps:

[0111] S401: When it is determined according to the original identifier that no initial deduplication metadata exists in the index pool, the data block is stored in the data pool.

[0112] S402: Generate initial deduplication metadata corresponding to the data block.

[0113] S403: Generate an original identifier of the data block according to the initial deduplication metadata.

[0114] S404: Generate a specific identifier corresponding to the initial deduplication metadata according to the original identifier.

[0115] S405: Store the original identifier, the initial deduplication metadata, and the specific identifier in the index pool in correspondence.

[0116] In some embodiments, the data block is first stored in the data pool according to the segmentation information used to indicate the storage of the data block in the storage system, and the segmentation information includes the name of the target object, the file name corresponding to the data block, the segmentation number of the data block, and the size of the data block. Then, according to the data pool identifier of the storage data block, the name of the storage bucket, the name of the target object, the file name corresponding to the data block, the summary value, the storage location of the file corresponding to the data block in the data pool, and the size of the data block, the initial deduplication metadata is generated, and the original identifier of the initial deduplication metadata is generated with reference to the method shown in formula (1). Then, the specific identifier of the initial deduplication metadata is generated with reference to the method shown in formula (2). Finally, the original identifier, the initial deduplication metadata, and the specific identifier are stored in the index pool accordingly, so that the initial deduplication metadata can be retrieved later.

[0117] In the disclosed embodiment, when all data blocks of the target object are stored, object metadata is generated according to the target object; and the object metadata is stored in the index pool.

[0118] After all data blocks of the target object are stored, object metadata is generated based on the name of the target object, the information record table, the data pool storing the target object, and the size of the target object. The object metadata is then stored in the index pool. By searching the object metadata in the index pool, the complete data of the target object can be obtained.

[0119] In some embodiments, the implementation process of the deduplication data processing method can also refer to Figure 5 The specific steps are as follows:

[0120] S501, deduplication initialization. The initialization process is executed when the client starts to upload a new target object. Specifically, when the client transmits the first data block, the storage system generates a segmentation information, which includes the target object name, the file name corresponding to the data block, the segment number (starting from 1), the segment size and other information. At the same time, the storage system also generates an array to save the segmentation information of each data block in the target object, and declares a variable offset = 0 to indicate the starting position of the first data block written this time.

[0121] S502, determine whether the data block is duplicate data. Calculate the MD5 value of the data block transmitted this time by the MD5 algorithm, and determine the index pool and storage node where the deduplication metadata of the data block is located according to the MD5 value, and determine the name of the data pool used to store the data block according to the name of the storage bucket corresponding to the data block. Based on the above information, calculate the original identifier of the deduplication metadata with reference to the method shown in formula (1), and query whether there is deduplication metadata in the index pool according to the original identifier. If there is deduplication metadata, it means that the data block is duplicate data; if there is no deduplication metadata, it means that the data block is not duplicate data.

[0122] S503a, for the data block determined to be duplicate data, the data block is no longer written into the data pool, and the attribute information of the data block is updated. Specifically, new segmentation information is first generated based on the deduplication metadata, and the segmentation information is written into the array. Then, a new deduplication metadata is generated based on the name of the target object, the name of the storage bucket, and the deduplication metadata, and a specific identifier is generated according to the method described in formula (2). Finally, the offset is reset, and a temporary segmentation information is generated. When the next data block is received, the temporary segmentation information is changed according to the information of the next data block.

[0123] S503b, for a data block of non-duplicate data, the data block needs to be written into the data pool according to the segmentation information of the data block. After being written into the data pool, deduplication metadata of the data block is generated, and an original identifier is generated according to formula (1), and a specific identifier is generated according to formula (2), and the deduplication metadata, the original identifier and the specific identifier are written into the index pool. Finally, the segmentation information is updated according to the information of the next data block.

[0124] S504, receiving the next data block, and repeating steps S502 and S503 until the target object is completely stored.

[0125] S505: After the target object is stored, generate object metadata of the target object, and store the object metadata in the index pool.

[0126] It should be noted that if the last data block is not duplicate data, the segmentation information corresponding to the last data block needs to be stored in the array.

[0127] By splitting the target object into data blocks for storage and deduplicating the data when the same data blocks exist in the data pool, that is, not uploading duplicate data blocks to the data pool, the uploading efficiency of the data object is improved, and at the same time, writing duplicate data blocks to the data pool is avoided, saving the storage space of the data pool, thereby improving the utilization of the storage space in the storage system.

[0128] Corresponding to the above-mentioned embodiment of the method for processing duplicate data, the present disclosure also provides an embodiment of an apparatus for processing duplicate data.

[0129] Please refer to Figure 6 A block diagram of a repeated data processing device is shown, and the repeated data processing device may include the following parts:

[0130] An acquiring unit 601 is configured to acquire a data block of a target object uploaded by a client in a single upload, wherein the data block is a part of the object data of the target object;

[0131] A calculation unit 602 is configured to calculate an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in the data pool;

[0132] The first determination unit 603 is used to determine that the data block is duplicate data, no longer store the data block in the data pool, and update the attribute information of the data block when it is determined that the initial deduplication metadata exists in the index pool based on the original identifier; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

[0133] By splitting the target object into data blocks for storage and deduplicating the data when the same data blocks exist in the data pool, that is, not uploading duplicate data blocks to the data pool, the uploading efficiency of the data object is improved, and at the same time, writing duplicate data blocks to the data pool is avoided, saving the storage space of the data pool, thereby improving the utilization of the storage space in the storage system.

[0134] In the embodiment of the present disclosure, the computing unit 602 is used to:

[0135] Calculate the digest value corresponding to the data block according to the digest algorithm;

[0136] Calculate the index pool and storage node where the initial deduplication metadata corresponding to the data block is located according to the summary value;

[0137] Determine the name of the data pool according to the name of the storage bucket corresponding to the data block;

[0138] The original identifier is calculated according to the name of the data pool, the storage node and the digest value.

[0139] In the embodiment of the present disclosure, the device further includes:

[0140] A first generating unit is configured to generate, when the data block is a starting data block of the target object, an information recording table of the target object and segmentation information of the data block, wherein the information recording table is used to record segmentation information of all data blocks of the target object, and the segmentation information is used to indicate that the data block is stored in the data pool;

[0141] The writing unit is used to write the segmentation information of the data block into the information recording table.

[0142] In the embodiment of the present disclosure, the device further includes:

[0143] A second determining unit is used to determine, when the data block is a starting data block of the target object, that a storage position code corresponding to a storage position of the data block in the data pool is a starting position code;

[0144] The third determining unit is configured to determine the storage position of the current data block according to the storage position of the previous data block in the data pool when the data block is a non-starting data block of the target object.

[0145] In the embodiment of the present disclosure, the first determining unit 603 is used to:

[0146] Generate new deduplication metadata according to the name of the target object, the name of the storage bucket corresponding to the data block, and the initial deduplication metadata;

[0147] Generate a specific identifier corresponding to the new deduplication metadata according to the original identifier, wherein the specific identifier is used to indicate the amount of deduplication metadata in the index pool;

[0148] The new deduplication metadata and the specific identifier are stored in the index pool in correspondence, wherein the number of the deduplication metadata in the index pool is used to indicate the number of times the data block is referenced.

[0149] In the embodiment of the present disclosure, the device further includes:

[0150] a first storage unit, configured to store the data block in the data pool when it is determined according to the original identifier that no initial deduplication metadata exists in the index pool;

[0151] A second generating unit, configured to generate initial deduplication metadata corresponding to the data block;

[0152] A third generating unit, configured to generate an original identifier of the data block according to the initial deduplication metadata;

[0153] a fourth generating unit, configured to generate a specific identifier corresponding to the initial deduplication metadata according to the original identifier;

[0154] The second storage unit is used to store the original identifier, the initial deduplication metadata and the specific identifier in the index pool in correspondence.

[0155] In the embodiment of the present disclosure, the device further includes:

[0156] A fifth generating unit, configured to generate object metadata according to the target object when all data blocks of the target object are stored;

[0157] The third storage unit is used to store the object metadata in the index pool.

[0158] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0160] The embodiment of the apparatus for processing duplicate data disclosed in the present invention can be applied to a computer device. The apparatus embodiment can be implemented by software, or by hardware, or by a combination of software and hardware. Taking software implementation as an example, as a logically sensed apparatus, the processor of the computer device in which it is located reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 7 The figure is a hardware structure diagram of a computer device where the repeated data processing device of the present disclosure is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device where the apparatus in the embodiment is located may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0161] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0162] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0163] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, just to name a few.

[0164] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0165] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0166] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0167] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0168] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for processing duplicate data, characterized in that: The method comprises: Acquire a data block of a target object uploaded by a client in a single time, wherein the data block is a part of the object data of the target object; Calculating an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in the data pool; When it is determined based on the original identifier that the initial deduplication metadata exists in the index pool, the data block is determined to be duplicate data, the data block is no longer stored in the data pool, and the attribute information of the data block is updated; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

2. The method according to claim 1, characterized in that: The calculating the original identifier corresponding to the data block includes: Calculate the digest value corresponding to the data block according to the digest algorithm; Calculating a storage node of initial deduplication metadata corresponding to the data block according to the summary value; Determine the name of the data pool according to the name of the storage bucket corresponding to the data block; The original identifier is calculated according to the name of the data pool, the storage node and the digest value.

3. The method according to claim 1, characterized in that The method further comprises: In the case where the data block is a starting data block of the target object, generating an information recording table of the target object and segmentation information of the data block, wherein the information recording table is used to record the segmentation information of all data blocks of the target object, and the segmentation information is used to indicate that the data block is stored in the data pool; The segmentation information of the data block is written into the information recording table.

4. The method according to claim 1, characterized in that The method further comprises: In a case where the data block is a starting data block of the target object, determining a storage position code corresponding to a storage position of the data block in the data pool as a starting position code; In the case that the data block is a non-starting data block of the target object, the storage position of the current data block is determined according to the storage position of the previous data block in the data pool.

5. The method according to claim 1, characterized in that The updating of the attribute information of the data block includes: generating new deduplication metadata according to the name of the target object, the name of the storage bucket corresponding to the data block, and the initial deduplication metadata; Generate a specific identifier corresponding to the new deduplication metadata according to the original identifier, wherein the number of the specific identifiers is used to represent the number of deduplication metadata in the index pool; The new deduplication metadata and the specific identifier are stored in the index pool in correspondence, wherein the number of the deduplication metadata in the index pool is used to indicate the number of times the data block is referenced.

6. The method according to claim 1, characterized in that After calculating the original identifier of the initial deduplication metadata corresponding to the data block, the method further includes: When it is determined according to the original identifier that there is no initial deduplication metadata in the index pool, storing the data block in the data pool; Generating initial deduplication metadata corresponding to the data block; generating an original identifier of the data block according to the initial deduplication metadata; Generating a specific identifier corresponding to the initial deduplication metadata according to the original identifier; The original identifier, the initial deduplication metadata and the specific identifier are stored in the index pool in correspondence.

7. The method according to claim 1, characterized in that The method further comprises: When all data blocks of the target object are stored, generating object metadata according to the target object; The object metadata is stored in the index pool.

8. A device for processing duplicate data, characterized in that: The device comprises: An acquiring unit, configured to acquire a data block of a target object uploaded by a client in a single time, wherein the data block is a part of the object data of the target object; a calculation unit, configured to calculate an original identifier of initial deduplication metadata of the data block, wherein the initial deduplication metadata is used to indicate whether the data block exists in the data pool; The first determination unit is used to determine that the data block is duplicate data, no longer store the data block in the data pool, and update the attribute information of the data block when it is determined that the initial deduplication metadata exists in the index pool based on the original identifier; the index pool is used to store the initial deduplication metadata, and the data pool is used to store the object data of the target object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When executing the program, the processor implements the steps of the method according to any one of claims 1 to 7.

Images