Data object deduplication processing method and device, equipment and storage medium

Abstract

The invention relates to a data object deduplication processing method and device, equipment and a storage medium. The method comprises the following steps: in response to a deduplication processing request of a target data object, obtaining a candidate data object with the same fingerprint as the target data object in a storage system; and if the target data object is the same as the candidate data object, performing deduplication processing on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object. When it is determined that the fingerprints of the target data object and the candidate data object are the same, the comparison process of the data objects is added to accurately judge whether the contents of the two data objects are consistent or not, and the situation that the data contents are different while the fingerprints are the same is prevented, and data deduplication errors are caused; moreover, when the target data object is the same as the candidate data object, the unique identifier of the fingerprint is introduced, and the target data object is subjected to deduplication processing according to the unique identifier, that is, the accuracy of data deduplication is further ensured through the unique identifier, and data loss caused by data deduplication errors is prevented.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data object deduplication processing method, device, equipment and storage medium. Background Art

[0002] In the field of storage technology, with the ever-increasing amount of data, when the same data appears, deduplication is performed to save only one copy of the data, which can save storage space.

[0003] A fingerprint is usually generated for each data object in the storage system. The fingerprint can be a fixed-length, unique value or string generated by processing the original data (such as text, image, audio or video files) with a specific algorithm, which can ensure the information integrity and uniqueness of the data object. In related technologies, when it is necessary to perform deduplication processing on a data object, the fingerprint library can be searched according to the fingerprint of the data object. If the same fingerprint exists, one of the two data objects with the same fingerprint can be deleted at random.

[0004] However, in the related art, when deduplicating data objects, deletion errors are prone to occur, resulting in data loss. Summary of the invention

[0005] Based on this, it is necessary to provide a data object deduplication processing method, device, equipment and storage medium to address the above technical problems, so as to prevent data deletion errors and data loss.

[0006] In a first aspect, the present application provides a data object deduplication processing method, comprising:

[0007] In response to a deduplication processing request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system;

[0008] If the target data object is the same as the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0009] In the data object deduplication processing method provided by the embodiment of the present application, by responding to the deduplication processing request of the target data object, a candidate data object with the same fingerprint as the target data object is obtained in the storage system. If the target data object is the same as the candidate data object, the target data object is deduplicated according to the unique identifier corresponding to the fingerprint of the candidate data object. In this method, when deduplicating the target data object, the candidate data object with the same fingerprint as the target data object is first obtained in the storage system, and then it is determined whether the target data object is the same as the candidate data object, that is, when it is determined that the fingerprint of the target data object is the same as the candidate data object, a comparison process of the data object is also added to accurately determine whether the contents of the two data objects are completely consistent, to prevent the situation where the fingerprints are the same but the data contents are different, resulting in data deduplication errors; and, when the target data object is the same as the candidate data object, a unique identifier corresponding to the fingerprint is also introduced, and the target data object is deduplicated according to the unique identifier, that is, the accuracy of data deduplication is further ensured by the unique identifier, to prevent data changes, data deduplication errors, and data loss.

[0010] In one embodiment, obtaining a candidate data object having the same fingerprint as a target data object in a storage system includes:

[0011] Perform fingerprint calculation on the target data object to obtain a target fingerprint of the target data object;

[0012] Compare the target fingerprint with multiple fingerprints in the fingerprint library;

[0013] In the case that there is a fingerprint identical to the target fingerprint in the fingerprint database, the fingerprint identical to the target fingerprint is obtained, and a data object corresponding to the fingerprint identical to the target fingerprint in the storage system is determined as a candidate data object.

[0014] In the data object deduplication processing method provided by the embodiment of the present application, fingerprint calculation is performed on the target data object to obtain the target fingerprint of the target data object, and then the target fingerprint is compared with multiple fingerprints in the fingerprint library. When there is a fingerprint identical to the target fingerprint in the fingerprint library, the fingerprint identical to the target fingerprint is obtained, and the data object corresponding to the fingerprint identical to the target fingerprint in the storage system is determined as a candidate data object. In this method, the target fingerprint of the target data object is obtained by fingerprint calculation, and the target fingerprint is compared with multiple fingerprints in the fingerprint library to obtain the fingerprint identical to the target fingerprint from the fingerprint library, so that the server can quickly and accurately obtain the candidate data object from the storage system based on the identical fingerprint, thereby improving the accuracy of obtaining the candidate data object.

[0015] In one embodiment, the method further comprises:

[0016] If there is no fingerprint identical to the target fingerprint in the fingerprint database, the target fingerprint is added to the fingerprint database.

[0017] In the data object deduplication processing method provided in the embodiment of the present application, when there is no fingerprint identical to the target fingerprint in the fingerprint library, the target fingerprint is added to the fingerprint library. In this method, when there is no fingerprint identical to the target fingerprint in the fingerprint library, it means that the target data object is a new data object, and the target data object needs to be stored in the storage system, and the target fingerprint of the target data object is added to the fingerprint library, providing fingerprint verification support for the deduplication processing of other data objects.

[0018] In one embodiment, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object, including:

[0019] When the target data object and the candidate data object are compared byte by byte, a reference unique identifier corresponding to the data object with the same fingerprint as the target data object in the current fingerprint database is obtained;

[0020] According to the unique identifier and the reference unique identifier, update verification is performed on the object content corresponding to the same fingerprint to obtain an update verification result;

[0021] Based on the update verification result, the target object is deduplicated.

[0022] In the data object deduplication processing method provided by the embodiment of the present application, when the byte-by-byte comparison between the target data object and the candidate data object is completed, the reference unique identifier corresponding to the data object with the same fingerprint as the target data object in the current fingerprint library is obtained, and then the object content corresponding to the same fingerprint is updated and verified based on the unique identifier and the reference unique identifier to obtain an update verification result, and then the target data object is deduplicated based on the update verification result. In this method, an optional way of deduplication processing is provided; by obtaining the reference unique identifier corresponding to the fingerprint with the same fingerprint as the target data object in the fingerprint library after the byte-by-byte comparison is completed, based on the reference unique identifier and the unique identifier, the object content corresponding to the same fingerprint is verified for content changes before and after the byte-by-byte comparison, so as to perform deduplication processing on the target data object according to the verification result, improve the accuracy of data object deduplication, and prevent data deduplication errors.

[0023] In one embodiment, the object content corresponding to the same fingerprint is updated and verified according to the unique identifier and the reference unique identifier to obtain an update verification result, including:

[0024] comparing the unique identifier to a reference unique identifier;

[0025] If the unique identifier is consistent with the reference unique identifier, it is determined that the updated verification result is that the object content corresponding to the same fingerprint has not changed;

[0026] If the unique identifier is inconsistent with the reference unique identifier, it is determined that the updated verification result is that the object content corresponding to the same fingerprint has changed.

[0027] In the data object deduplication processing method provided in the embodiment of the present application, the unique identifier is compared with the reference unique identifier. If the unique identifier is consistent with the reference unique identifier, the update verification result is determined to be that the object content corresponding to the same fingerprint has not changed. If the unique identifier is inconsistent with the reference unique identifier, the update verification result is determined to be that the object content corresponding to the same fingerprint has changed. In this method, an optional way to quickly determine the update verification result is provided; by comparing the unique identifier with the reference unique identifier, it is determined whether the object content corresponding to the same fingerprint has changed before and after the byte-by-byte comparison, thereby obtaining the update verification result, which provides data support for the deduplication processing of the target data object.

[0028] In one embodiment, deduplication processing is performed on the target data object according to the update verification result, including:

[0029] If the update verification result is that the object content corresponding to the same fingerprint has not changed, the target data object is deleted;

[0030] If the update verification result is that the object content corresponding to the same fingerprint has changed, a byte-by-byte comparison operation is re-initiated for the target data object, and deduplication processing is performed on the target data object based on the comparison result.

[0031] In the data object deduplication processing method provided by the embodiment of the present application, if the update verification result is that the object content corresponding to the same fingerprint has not changed, the target data object is deleted. If the update verification result is that the object content corresponding to the same fingerprint has changed, the byte-by-byte comparison operation is re-initiated for the target data object, and the target data object is deduplicated according to the comparison result. In this method, when it is determined that the object content of the same fingerprint has not changed before and after the byte-by-byte comparison, the target data object is deleted. When it is determined that the object content of the same fingerprint has changed before and after the byte-by-byte comparison, the target data object is not deduplicated, but a byte-by-byte comparison operation is re-initiated to further determine whether the target data object can be deduplicated, thereby improving the accuracy of data object deduplication processing and preventing data loss.

[0032] In one embodiment, the method further comprises:

[0033] Comparing the target data object with the candidate data object byte by byte;

[0034] In the case where each byte of the target data object is identical to each byte of the candidate data object, determining that the target data object is identical to the candidate data object;

[0035] In the case that there is at least one byte difference between the target data object and the candidate data object, it is determined that the target data object is not the same as the candidate data object.

[0036] In the data object deduplication processing method provided by the embodiment of the present application, the target data object is compared byte by byte with the candidate data object. When each byte of the target data object is the same as each byte of the candidate data object, the target data object is determined to be the same as the candidate data object. When there is at least one byte different between the target data object and the candidate data object, the target data object is determined to be different from the candidate data object. In this method, an optional way to quickly determine whether the target data object is the same as the candidate data object is provided; by comparing the target data object with the candidate data object byte by byte, it can be determined whether each byte of the target data object is the same as each byte of the candidate data object, thereby judging whether the target data object and the candidate data object are the same. If each byte of the target data object is the same as each byte of the candidate data object, the target data object is the same as the candidate data object. If there is a byte in the target data object that is different from the byte of the candidate data object, the target data object is different from the candidate data object; by comparing the target data object and the candidate data object byte by byte, it is determined whether the contents of the two data objects are the same, providing a guarantee for accurate deduplication processing of the target data object.

[0037] In a second aspect, the present application further provides a data object deduplication processing device, comprising:

[0038] A candidate object acquisition module, configured to respond to a deduplication processing request of a target data object and acquire a candidate data object having the same fingerprint as the target data object in the storage system;

[0039] The object deduplication processing module is used to perform deduplication processing on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object if the target data object is the same as the candidate data object.

[0040] In a third aspect, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any one of the embodiments of the first aspect when executing the computer program.

[0041] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in any one of the embodiments of the first aspect are implemented.

[0042] In a fifth aspect, the embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in any one of the embodiments in the first aspect are implemented.

[0043] The above-mentioned data object deduplication processing method, device, equipment and storage medium obtains a candidate data object with the same fingerprint as the target data object in the storage system in response to a deduplication processing request of the target data object. If the target data object is the same as the candidate data object, the target data object is deduplicated according to the unique identifier corresponding to the fingerprint of the candidate data object. In this method, when deduplicating the target data object, the candidate data object with the same fingerprint as the target data object in the storage system is first obtained, and then it is determined whether the target data object is the same as the candidate data object. That is, when it is determined that the fingerprint of the target data object is the same as that of the candidate data object, a comparison process of the data object is also added to accurately determine whether the contents of the two data objects are completely consistent, so as to prevent the situation where the fingerprints are the same but the data contents are different, resulting in data deduplication errors; and, when the target data object is the same as the candidate data object, the unique identifier corresponding to the fingerprint is also introduced, and the target data object is deduplicated according to the unique identifier, that is, the accuracy of data deduplication is further ensured by the unique identifier, so as to prevent data changes, data deduplication errors, and data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] FIG1 is a diagram showing an application environment of a data object deduplication processing method according to an embodiment;

[0046] FIG2 is a schematic diagram of a flow chart of a data object deduplication processing method in one embodiment;

[0047] FIG3 is a schematic diagram of a process for obtaining candidate data objects in one embodiment;

[0048] FIG4 is a schematic diagram of a flow chart of byte-by-byte comparison in one embodiment;

[0049] FIG5 is a schematic diagram of a flow chart of deduplication processing in one embodiment;

[0050] FIG6 is a schematic diagram of a process for determining an update verification result in one embodiment;

[0051] FIG7 is a schematic diagram of a flow chart of deduplication processing in another embodiment;

[0052] FIG8 is a schematic diagram of a flow chart of a data object deduplication processing method in another embodiment;

[0053] FIG9 is a schematic diagram of the structure of a data object deduplication processing device in one embodiment;

[0054] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The data object deduplication processing method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Optionally, the server 104 can be a deduplication processing system. The terminal 102 is used to trigger a deduplication processing request for the target data object. For example, the user can upload the target data object to the server 104 in the terminal 102, and the deduplication processing request for the target data object is triggered when the target data object is uploaded. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0057] The technical background of this application is described below.

[0058] In the field of storage technology, the ever-increasing amount of data requires the continuous expansion of storage devices, which will increase the storage costs of enterprises. When the same data appears, deduplication is performed and only one copy of the data is saved to save storage space.

[0059] A fingerprint is usually generated for each data object in the storage system. The fingerprint can be a fixed-length, unique value or string generated by processing the original data (such as text, image, audio or video file) with a specific algorithm, which can ensure the information integrity and uniqueness of the data object. In the related art, in the object storage system, deduplication processing can be performed at the granularity of the object. According to different fingerprint algorithms, a fingerprint is generated for each data object. Common fingerprint algorithms include signature algorithms such as Message-Digest Algorithm 5 (MD5) or hash algorithms. When it is necessary to perform deduplication processing on a data object, the fingerprint library will be checked according to the fingerprint value of the object: if the corresponding fingerprint is not found in the fingerprint library, it means that there must be no object with the same content as the current object, and a new fingerprint is added. If the corresponding fingerprint can be found in the fingerprint library, it means that the content of the existing object is the same as the content of the current object, and the reference count of the corresponding fingerprint is increased. When the user has the need to delete the object, the status of the corresponding fingerprint in the fingerprint library needs to be modified. If the reference count of the fingerprint is equal to 1, it means that except the current object, no other object references the current fingerprint, and the corresponding fingerprint in the fingerprint library is deleted; if the reference count of the fingerprint is greater than 1, it means that there are other objects that reference the current fingerprint, and the reference count of the corresponding fingerprint is decremented.

[0060] However, the fact that the fingerprints of two objects are the same does not guarantee that the data of the two objects are necessarily the same. This is determined by the nature of the fingerprint algorithm MD5 or the hash signature algorithm. In this way, when deduplicating data objects, deletion errors are prone to occur, resulting in data loss.

[0061] Based on this, an embodiment of the present application provides a data object deduplication processing method. When deduplication processing is performed on a target data object, a candidate data object with the same fingerprint as the target data object is first obtained in the storage system, and then it is determined whether the target data object and the candidate data object are the same. That is, when it is determined that the fingerprint of the target data object and the candidate data object are the same, a data object comparison process is added to accurately determine whether the contents of the two data objects are completely consistent, so as to prevent the situation where the fingerprints are the same but the data contents are different, resulting in data deduplication errors; moreover, when the target data object is the same as the candidate data object, a unique identifier corresponding to the fingerprint is also introduced, and the target data object is deduplicated according to the unique identifier, that is, the accuracy of data deduplication is further ensured by the unique identifier, so as to prevent data changes from causing data deduplication errors and causing data loss.

[0062] In an exemplary embodiment, as shown in FIG2 , a data object deduplication processing method is provided, and the method is applied to the server 104 in FIG1 as an example for description, including the following steps 201 to 202. Among them:

[0063] S201 , in response to a deduplication request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system.

[0064] A data object is usually a collection of multiple data elements, which can be records, files, images, audio or video. For example, a record in a database can be considered a data object. Deduplication, also known as data de-duplication, is a data management technology that aims to reduce the storage capacity used in a storage system.

[0065] The deduplication request refers to a request triggered when a target data object needs to be deduplicated in a deduplication application of the server. Naturally, after receiving the deduplication request, the server needs to deduplication the target data object.

[0066] In one embodiment, the request is triggered by the target user through the server interface in the deduplication processing application; wherein, the triggering method includes but is not limited to external input device clicks, voice control, touch screen triggering, etc., and the embodiment of the present application does not limit the triggering method.

[0067] In another embodiment, the request is triggered according to a preset automatic program; for example, if the server receives the target data object transmitted by the user or the storage system, the deduplication processing request of the target data object is automatically triggered according to the preset program; or, after the server receives the target data object transmitted by the user or the storage system, if the size of the received target data object reaches a certain threshold, and / or, when the application is in an idle state, the deduplication processing request of the target data object is automatically triggered according to the preset program. The embodiment of the present application does not limit the triggering method of the deduplication processing request. Among them, the target data object transmitted by the user can be a data object that the user wants to store in the storage system. When the user wants to store the target data object in the storage system, the target data object is uploaded to the server. After receiving the uploaded target data object, the server determines whether there is the same data object in the storage system. If so, the target data object is deduplicated. If not, the target data object can be stored in the storage system. The target data object transmitted by the storage system can be any data object already stored in the storage system. For example, the storage system will perform deduplication processing on the data objects stored in the system at regular intervals. When deduplication processing is required, the storage system can transmit the target data object to the server.

[0068] Fingerprint refers to a fixed-length, unique value or string generated by processing a data object according to a fingerprint algorithm. Each data object corresponds to a fingerprint. Among them, fingerprint algorithms can include MD5 algorithms and hash algorithms. Candidate data objects refer to data objects that have the same fingerprint as the target data object.

[0069] Exemplarily, a method for obtaining a candidate data object having the same fingerprint as a target data object in a storage system may be to perform fingerprint calculation on the target data object according to a preset fingerprint algorithm to obtain a target fingerprint of the target data object, and then compare the target fingerprint with the fingerprint of each data object in the storage system; if the fingerprint of a data object is the same as the target fingerprint, the data object is determined as a candidate data object, and the candidate data object is obtained from the storage system.

[0070] The fingerprint of each data object in the storage system may be obtained by processing according to a preset fingerprint algorithm when the candidate data object is currently acquired, or may be obtained by processing according to a preset fingerprint algorithm at a historical time and stored in the database.

[0071] S202: If the target data object is identical to the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0072] ‌‌Unique identification refers to an identifier used to uniquely identify the fingerprint of each data object. The unique identification of the fingerprint ensures that each fingerprint has a unique identity in the system, avoiding confusion and duplication. The unique identification can be numbers, letters, symbols, etc., or any combination of numbers, letters, symbols, or other forms of encoding. ‌

[0073] In the embodiment of the present application, the unique identifier can be a monotonically increasing integer, occupying 8 bytes. Each time a data object is stored in the storage system, a fingerprint is calculated for each data object, and a unique identifier is generated for each fingerprint. Even when a data object and the fingerprint of the data object are deleted, the unique identifier corresponding to the data object still exists. When a new unique identifier is generated, it is also generated on this basis, and the unique identifier corresponding to the data object will not be deleted. In this way, it can be guaranteed that the unique identifier will not be repeated. For example, the generated unique identifiers can be saved to the disk. When the process is restarted or the device is restarted, the used unique identifier can be restored according to the disk record. The subsequent process is based on the existing unique identifier, monotonically increasing, and a new unique identifier is generated to ensure that the unique identifiers corresponding to the fingerprints in the entire system will not be repeated.

[0074] After the candidate data object is acquired in the storage system, the target data object is compared with the candidate data object to determine whether the target data object is the same as the candidate data object.

[0075] For example, each element in the target data object and the candidate data object may be compared one by one to determine whether the target data object is the same as the candidate data object. When each element in the target data object and the candidate data object is the same, it is determined that the target data object and the candidate data object are the same; when at least one element in the target data object and the candidate data object is different, it is determined that the target data object and the candidate data object are different.

[0076] If the target data object is the same as the candidate data object, then the unique identifier corresponding to the fingerprint of the candidate data object may be obtained, and deduplication processing may be performed on the target data object according to the unique identifier.

[0077] In one embodiment, the unique identifier corresponding to the fingerprint of the candidate data object may be obtained by obtaining the unique identifier corresponding to the fingerprint of the candidate data object according to the fingerprint identifier mapping relationship. For example, the fingerprint identifier mapping relationship stores the association relationship between multiple fingerprints and the unique identifiers corresponding to each fingerprint. When it is necessary to obtain the unique identifier corresponding to the fingerprint of the candidate data object, the fingerprint of the candidate data object may be matched with the multiple fingerprints in the above mapping relationship, and the unique identifier corresponding to the matched fingerprint may be determined as the unique identifier corresponding to the fingerprint of the candidate data object.

[0078] In another embodiment, the unique identifier corresponding to the fingerprint of the candidate data object may be obtained from an identifier database. For example, the identifier database stores the unique identifier corresponding to the fingerprint of each data object in the storage system. When it is necessary to obtain the unique identifier corresponding to the fingerprint of the candidate data object, the unique identifier corresponding to the candidate data object may be matched from the identifier database according to the candidate data object, and the unique identifier may be determined as the unique identifier corresponding to the fingerprint of the candidate data object.

[0079] After obtaining the unique identifier corresponding to the fingerprint of the candidate data object, obtain the unique identifier corresponding to the fingerprint identical to the target data object's fingerprint among all current unique identifiers, and then compare the unique identifier with the unique identifier corresponding to the fingerprint of the candidate data object. If the two unique identifiers are the same, the target data object is deleted. If the two unique identifiers are different, the target data object is not deleted. The target data object can be compared again with the data object with the same fingerprint as the target data object among all current data objects to determine whether the two data objects are the same.

[0080] When the fingerprints of the target data object and the candidate data object are the same, and the target data object and the candidate data object are the same, it is necessary to further confirm whether the content of the data object corresponding to the same fingerprint has changed during the comparison of the target data object and the candidate data object. If changed, it may be that during the comparison of the target data object and the candidate data object, the user deleted the candidate data object and added a new data object in the storage system, and the fingerprint of the new data object is the same as the fingerprint of the target data object.

[0081] In order to ensure that the data corresponding to the same fingerprint has not changed during the comparison process and to prevent data deletion errors, the unique identifier corresponding to the data object with the same fingerprint as the target data object can be obtained after comparison. If the unique identifier has not changed after comparison, it means that the content of the data object corresponding to the same fingerprint has not changed. At this time, the target data object can be deleted, and the storage system still stores data objects with the same content as the target data object. If the unique identifier has changed after comparison, it means that the content of the data object corresponding to the same fingerprint has also changed. At this time, the target data object cannot be deleted directly, because it cannot be guaranteed that the content of the two data objects with the same fingerprint is exactly the same. If only the fingerprint is the same, but the content is different, and the target data object is deleted, there will be no data object with the same content as the target data object in the storage system, which will lead to data deletion errors and cause data loss of the target data object.

[0082] In the data object deduplication processing method provided by the embodiment of the present application, by responding to the deduplication processing request of the target data object, a candidate data object with the same fingerprint as the target data object is obtained in the storage system. If the target data object is the same as the candidate data object, the target data object is deduplicated according to the unique identifier corresponding to the fingerprint of the candidate data object. In this method, when deduplicating the target data object, the candidate data object with the same fingerprint as the target data object is first obtained in the storage system, and then it is determined whether the target data object is the same as the candidate data object, that is, when it is determined that the fingerprint of the target data object is the same as the candidate data object, a comparison process of the data object is also added to accurately determine whether the contents of the two data objects are completely consistent, to prevent the situation where the fingerprints are the same but the data contents are different, resulting in data deduplication errors; and, when the target data object is the same as the candidate data object, a unique identifier corresponding to the fingerprint is also introduced, and the target data object is deduplicated according to the unique identifier, that is, the accuracy of data deduplication is further ensured by the unique identifier, to prevent data changes, data deduplication errors, and data loss.

[0083] Based on the above embodiment, an embodiment is provided to illustrate the process of obtaining a candidate data object having the same fingerprint as the target data object in the storage system.

[0084] In an exemplary embodiment, as shown in FIG3 , obtaining a candidate data object having the same fingerprint as a target data object in a storage system includes:

[0085] S301, performing fingerprint calculation on a target data object to obtain a target fingerprint of the target data object.

[0086] Exemplarily, a preset fingerprint algorithm may be used to perform fingerprint calculation on the target data object to obtain a target fingerprint of the target data object. The fingerprint algorithm may include a signature algorithm such as MD5 or hash.

[0087] Taking the hash algorithm as an example, the target data object can be input into the hash algorithm, and the hash algorithm converts the input target data object into a hash value of a fixed length through a series of complex mathematical operations. The hash value refers to the target fingerprint of the target data object.

[0088] S302: Compare the target fingerprint with multiple fingerprints in the fingerprint database.

[0089] The fingerprint database refers to a database storing fingerprints of various data objects in the storage system, that is, the fingerprint database stores fingerprints of multiple data objects.

[0090] After obtaining the target fingerprint of the target data object, the target fingerprint is compared with multiple fingerprints in the fingerprint library to determine whether there is a fingerprint in the fingerprint library that is identical to the target fingerprint.

[0091] S303: if there is a fingerprint identical to the target fingerprint in the fingerprint database, obtain the fingerprint identical to the target fingerprint, and determine the data object corresponding to the fingerprint identical to the target fingerprint in the storage system as a candidate data object.

[0092] After comparing the target fingerprint with multiple fingerprints in the fingerprint library, if there is a fingerprint identical to the target fingerprint in the fingerprint library, the fingerprint identical to the target fingerprint is obtained from the fingerprint library, and based on the fingerprint, the data object corresponding to the fingerprint is obtained from the storage system, and the data object is the candidate data object.

[0093] The above is the case where the same fingerprint exists in the fingerprint library. When the fingerprint library does not contain the same fingerprint as the target fingerprint, it means that there must be no data object identical to the target data object in the storage system, and additional processing is required.

[0094] Based on this, in an exemplary embodiment, the method further includes:

[0095] If there is no fingerprint identical to the target fingerprint in the fingerprint database, the target fingerprint is added to the fingerprint database.

[0096] After comparing the target fingerprint with multiple fingerprints in the fingerprint library, if there is no fingerprint identical to the target fingerprint in the fingerprint library, then there is no data object identical to the target data object in the storage system. At this time, the target fingerprint of the target data object is added to the fingerprint library, and the target data object is stored in the storage system, that is, the target data object is new data and is only stored in the storage system without deduplication processing.

[0097] In this embodiment, when there is no fingerprint identical to the target fingerprint in the fingerprint library, the target fingerprint is added to the fingerprint library. In this method, when there is no fingerprint identical to the target fingerprint in the fingerprint library, it means that the target data object is a new data object, and the target data object needs to be stored in the storage system, and the target fingerprint of the target data object is added to the fingerprint library, providing fingerprint verification support for deduplication processing of other data objects.

[0098] In the data object deduplication processing method provided by the embodiment of the present application, fingerprint calculation is performed on the target data object to obtain the target fingerprint of the target data object, and then the target fingerprint is compared with multiple fingerprints in the fingerprint library. When there is a fingerprint identical to the target fingerprint in the fingerprint library, the fingerprint identical to the target fingerprint is obtained, and the data object corresponding to the fingerprint identical to the target fingerprint in the storage system is determined as a candidate data object. In this method, the target fingerprint of the target data object is obtained by fingerprint calculation, and the target fingerprint is compared with multiple fingerprints in the fingerprint library to obtain the fingerprint identical to the target fingerprint from the fingerprint library, so that the server can quickly and accurately obtain the candidate data object from the storage system based on the identical fingerprint, thereby improving the accuracy of obtaining the candidate data object.

[0099] Based on any of the above embodiments, an embodiment is provided to illustrate the process of determining whether the target data object is the same as the candidate data object.

[0100] In an exemplary embodiment, as shown in FIG4 , the method further includes:

[0101] S401, comparing the target data object with the candidate data object byte by byte.

[0102] In the embodiment of the present application, when determining whether the target data object and the candidate data object are the same, the target data object and the candidate data object are compared byte by byte.

[0103] Exemplarily, for each data object, traverse in byte order, starting from the start position of the data object, process each byte one by one until the entire data object is traversed, and then for each byte, compare it with the corresponding byte of another data object. If the two bytes are the same, continue to compare the next byte.

[0104] For example, for a target data object, starting from the start position of the target data object, traverse in byte order to obtain multiple target bytes of the target data object. Similarly, for a candidate data object, starting from the start position of the candidate data object, traverse in byte order to obtain multiple candidate bytes of the candidate data object. Then, compare the first target byte of the target data object with the first candidate byte of the candidate data object, and then compare the second target byte of the target data object with the second candidate byte of the candidate data object, and so on, until all target bytes and candidate bytes are compared.

[0105] S402: When each byte of the target data object is identical to each byte of the candidate data object, determine that the target data object is identical to the candidate data object.

[0106] After comparing the target data object with the candidate data object byte by byte, if each byte of the target data object is identical to each byte of the candidate data object, it is determined that the target data object and the candidate data object are identical, that is, the data contents of the target data object and the candidate data object are identical.

[0107] S403: When there is at least one byte difference between the target data object and the candidate data object, determine that the target data object is different from the candidate data object.

[0108] After comparing the target data object with the candidate data object byte by byte, if there is a byte in the target data object that is different from the byte of the candidate data object, it is determined that the target data object and the candidate data object are different, that is, the data content of the target data object and the candidate data object is different. At this time, the target data object is ignored, and the fingerprint of the target data object is not added to the fingerprint library, that is, the current target data object does not participate in the deduplication process, will not share data with other objects, and the target data object itself will not be deduplicated. For objects with the same fingerprint but different data, only one object's fingerprint is retained.

[0109] In the data object deduplication processing method provided by the embodiment of the present application, the target data object is compared byte by byte with the candidate data object. When each byte of the target data object is the same as each byte of the candidate data object, the target data object is determined to be the same as the candidate data object. When there is at least one byte different between the target data object and the candidate data object, the target data object is determined to be different from the candidate data object. In this method, an optional way to quickly determine whether the target data object is the same as the candidate data object is provided; by comparing the target data object with the candidate data object byte by byte, it can be determined whether each byte of the target data object is the same as each byte of the candidate data object, thereby judging whether the target data object and the candidate data object are the same. If each byte of the target data object is the same as each byte of the candidate data object, the target data object is the same as the candidate data object. If there is a byte in the target data object that is different from the byte of the candidate data object, the target data object is different from the candidate data object; by comparing the target data object and the candidate data object byte by byte, it is determined whether the contents of the two data objects are the same, providing a guarantee for accurate deduplication processing of the target data object.

[0110] Based on any of the above embodiments, an embodiment is provided to illustrate the process of performing deduplication processing on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0111] In an exemplary embodiment, as shown in FIG5 , deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object, including:

[0112] S501, when the target data object and the candidate data object are compared byte by byte, obtain a reference unique identifier corresponding to a data object with the same fingerprint as the target data object in the current fingerprint library.

[0113] The reference unique identifier refers to the unique identifier corresponding to the fingerprint in the current fingerprint library that is the same as the target fingerprint of the target data object after the byte-by-byte comparison is completed. The reference unique identifier may be the unique identifier corresponding to the fingerprint of the candidate data object, or it may be the unique identifier corresponding to the fingerprint newly added to the fingerprint library during the byte-by-byte comparison between the target data object and the candidate data object.

[0114] When the target data object and the candidate data object are compared byte by byte, the target fingerprint of the target data object is compared with the multiple fingerprints in the fingerprint library after the byte by byte comparison, and the unique identifier corresponding to the fingerprint in the current fingerprint library that is the same as the target fingerprint is obtained, and the unique identifier is determined as the reference unique identifier. The unique identifier corresponding to each fingerprint is also stored in the fingerprint library accordingly.

[0115] S502: Perform update verification on the object content corresponding to the same fingerprint according to the unique identifier and the reference unique identifier to obtain an update verification result.

[0116] The update verification refers to the content change verification of the object content corresponding to the same fingerprint before and after the byte-by-byte comparison. The update verification results include the object content corresponding to the same fingerprint has not changed and the object content corresponding to the same fingerprint has changed.

[0117] By using the unique identifier corresponding to the same fingerprint before byte-by-byte comparison and the unique identifier corresponding to the same fingerprint after byte-by-byte comparison, it is possible to determine whether the content of the data object corresponding to the same fingerprint has changed during the byte-by-byte comparison process, that is, to perform update verification on the object content corresponding to the same fingerprint, to verify whether the object content corresponding to the same fingerprint has changed before and after the byte-by-byte comparison, and to obtain an update verification result.

[0118] S503: De-duplication the target data object according to the update verification result.

[0119] After obtaining the update verification result, the target data object is deduplicated.

[0120] If the update verification result is that the object content corresponding to the same fingerprint has not changed, it means that the candidate data object and the fingerprint of the candidate data object have not been deleted. At this time, the target data object can be deduplicated and the candidate data object can be retained in the storage system.

[0121] If the update verification result is that the object content corresponding to the same fingerprint has changed, it means that the candidate data object has been deleted at the user's request, and the user has added a new data object to the storage system. The fingerprint of the new data object is the same as the fingerprint of the target data object. At this time, the target data object is not deduplicated.

[0122] In the data object deduplication processing method provided by the embodiment of the present application, when the byte-by-byte comparison between the target data object and the candidate data object is completed, the reference unique identifier corresponding to the data object with the same fingerprint as the target data object in the current fingerprint library is obtained, and then the object content corresponding to the same fingerprint is updated and verified based on the unique identifier and the reference unique identifier to obtain an update verification result, and then the target data object is deduplicated based on the update verification result. In this method, an optional way of deduplication processing is provided; by obtaining the reference unique identifier corresponding to the fingerprint with the same fingerprint as the target data object in the fingerprint library after the byte-by-byte comparison is completed, based on the reference unique identifier and the unique identifier, the object content corresponding to the same fingerprint is verified for content changes before and after the byte-by-byte comparison, so as to perform deduplication processing on the target data object according to the verification result, improve the accuracy of data object deduplication, and prevent data deduplication errors.

[0123] Based on the above embodiment, an embodiment is provided to illustrate the process of performing update verification on the object content corresponding to the same fingerprint according to the unique identifier and the reference unique identifier to obtain the update verification result.

[0124] In an exemplary embodiment, as shown in FIG6 , the object content corresponding to the same fingerprint is updated and verified according to the unique identifier and the reference unique identifier to obtain an update verification result, including:

[0125] S601, compare the unique identifier with a reference unique identifier.

[0126] After obtaining the reference unique identifier, the reference unique identifier is compared with the unique identifier to verify whether the object content corresponding to the same fingerprint has changed during the byte-by-byte comparison of the target data object and the candidate data object, so as to prevent data duplication errors.

[0127] S602: If the unique identifier is consistent with the reference unique identifier, it is determined that the updated verification result is that the object content corresponding to the same fingerprint has not changed.

[0128] The unique identifier is the unique identifier corresponding to the fingerprint identical to the target fingerprint of the target data object before the target data object and the candidate data object are compared byte by byte, and the reference unique identifier is the unique identifier corresponding to the fingerprint identical to the target fingerprint of the target data object after the target data object and the candidate data object are compared byte by byte. Based on the unique identifier before the byte by byte comparison and the unique identifier after the byte by byte comparison, it can be determined whether the object content corresponding to the same fingerprint before and after the byte by byte comparison has changed.

[0129] If the unique identifier is consistent with the reference unique identifier, the object content corresponding to the same fingerprint has not changed before and after the byte-by-byte comparison, and the updated verification result is that the object content corresponding to the same fingerprint has not changed.

[0130] S603: If the unique identifier is inconsistent with the reference unique identifier, it is determined that the object content corresponding to the same fingerprint has changed after the update verification result is obtained.

[0131] If the unique identifier is inconsistent with the reference unique identifier, the object content corresponding to the same fingerprint has changed before and after the byte-by-byte comparison, and the updated verification result is that the object content corresponding to the same fingerprint has changed.

[0132] In the data object deduplication processing method provided in the embodiment of the present application, the unique identifier is compared with the reference unique identifier. If the unique identifier is consistent with the reference unique identifier, the update verification result is determined to be that the object content corresponding to the same fingerprint has not changed. If the unique identifier is inconsistent with the reference unique identifier, the update verification result is determined to be that the object content corresponding to the same fingerprint has changed. In this method, an optional way to quickly determine the update verification result is provided; by comparing the unique identifier with the reference unique identifier, it is determined whether the object content corresponding to the same fingerprint has changed before and after the byte-by-byte comparison, thereby obtaining the update verification result, which provides data support for the deduplication processing of the target data object.

[0133] Based on the above embodiment, an embodiment is provided to illustrate the process of deduplicating the target data object according to the update verification result.

[0134] In an exemplary embodiment, as shown in FIG7 , deduplication processing is performed on the target data object according to the update verification result, including:

[0135] S701: If the update verification result is that the object content corresponding to the same fingerprint has not changed, the target data object is deleted.

[0136] When the update verification result is that the object content corresponding to the same fingerprint has not changed, it means that the user did not delete the candidate data object during the byte-by-byte comparison of the target data object and the candidate data object, that is, the object content corresponding to the same fingerprint has not changed before and after the byte-by-byte comparison. At this time, the target data object can be deleted and the candidate data object can be retained in the storage system. When the user wants to obtain the target data object subsequently, the candidate data object can be fed back to the user because the content of the candidate data object is exactly the same as that of the target data object.

[0137] S702: If the update verification result is that the object content corresponding to the same fingerprint has changed, a byte-by-byte comparison operation is re-initiated on the target data object, and deduplication processing is performed on the target data object according to the comparison result.

[0138] When the update verification result is that the object content corresponding to the same fingerprint has changed, it means that during the byte-by-byte comparison between the target data object and the candidate data object, the user deleted the candidate data object, and the fingerprint of the candidate data object was also deleted. At the same time, the user added a new data object into the storage system, and the fingerprint of the new data object is the same as the target fingerprint of the target data object. At this time, the target data object cannot be deleted directly because it cannot be ensured that the object content of the newly added data object is also the same as the object content of the target data object. In this case, a byte-by-byte comparison operation is re-initiated for the target data object, and the target data object is deduplicated according to the comparison result.

[0139] The re-initiated byte-by-byte comparison is to compare the target data object with the new data object byte-by-byte. If the target data object is the same as the new data object after the byte-by-byte comparison, it is determined that the content of the new data object is completely consistent with that of the target data object, and the target data object can be deleted and the new data object can be retained in the storage system. When the user wants to obtain the target data object later, the new data object can be fed back to the user.

[0140] If the target data object is different from the new data object after byte-by-byte comparison, it is determined that the new data object is different from the target data object, and there is no data object in the storage system with the same content as the target data object. At this time, the fingerprint of the target data object is added to the fingerprint library and the target data object is stored in the storage system.

[0141] In the data object deduplication processing method provided by the embodiment of the present application, if the update verification result is that the object content corresponding to the same fingerprint has not changed, the target data object is deleted. If the update verification result is that the object content corresponding to the same fingerprint has changed, the byte-by-byte comparison operation is re-initiated for the target data object, and the target data object is deduplicated according to the comparison result. In this method, when it is determined that the object content of the same fingerprint has not changed before and after the byte-by-byte comparison, the target data object is deleted. When it is determined that the object content of the same fingerprint has changed before and after the byte-by-byte comparison, the target data object is not deduplicated, but a byte-by-byte comparison operation is re-initiated to further determine whether the target data object can be deduplicated, thereby improving the accuracy of data object deduplication processing and preventing data loss.

[0142] In addition, in an exemplary embodiment, the present application also provides a data object deduplication processing method, as shown in FIG8 , which may include the following steps:

[0143] S801 , in response to a deduplication processing request of a target data object, fingerprint calculation is performed on the target data object to obtain a target fingerprint of the target data object.

[0144] S802 determines whether there is a fingerprint identical to the target fingerprint in the fingerprint database. If not, execute S803; if yes, execute S804.

[0145] S803, adding the target fingerprint to the fingerprint library.

[0146] S804: Acquire a fingerprint identical to the target fingerprint, and determine a data object corresponding to the fingerprint identical to the target fingerprint in the storage system as a candidate data object.

[0147] S805, compare the target data object with the candidate data object byte by byte to determine whether the target data object is the same as the candidate data object. If not, execute S810; if yes, execute S806.

[0148] S806: When the byte-by-byte comparison between the target data object and the candidate data object is completed, obtain a reference unique identifier corresponding to a data object with the same fingerprint as the target data object in the current fingerprint database.

[0149] S807, determine whether the unique identifier is consistent with the reference unique identifier. If yes, execute S808; if no, execute S809.

[0150] S808, delete the target data object.

[0151] S809, re-initiate a byte-by-byte comparison operation on the target data object, and perform deduplication processing on the target data object according to the comparison result.

[0152] S810, ignoring the target data object, and the target data object does not participate in the deduplication process.

[0153] The above process of S801-S810 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0154] In addition, in an exemplary embodiment, taking three data objects as an example, the process of deduplication processing of data objects after adding byte-by-byte comparison and unique identification is explained.

[0155] The fingerprints of the three data objects A, B, and C are the same, but only the contents of A and B are the same, and the content of C is different from that of A and B.

[0156] (1) The deduplication processing system processes data object A. There is no corresponding fingerprint in the fingerprint database, so the fingerprint of data object A is added to the fingerprint database.

[0157] (2) The deduplication processing system processes data object B and finds that there is a fingerprint in the fingerprint database that is the same as the fingerprint of data object B. The fingerprint corresponds to data object A, so A and B are compared byte by byte to prevent misjudgment.

[0158] (3) During the byte-by-byte comparison of A and B, the deduplication system deletes the fingerprint of data object A and then adds the fingerprint of data object C (because there is no corresponding fingerprint in the fingerprint library, the fingerprint of data object C is directly added to the fingerprint library, just like adding the fingerprint of data object A to the fingerprint library). After that, the deduplication system returns that the byte-by-byte comparison results of data object A and data object B are the same, and wants to increase the reference count of the fingerprint. However, the fingerprint in the current fingerprint library corresponds to the fingerprint of data object C, not the fingerprint of data object A; in fact, the contents of data object B and data object C are not the same. If deduplication is performed, it will cause data inconsistency errors in the storage system.

[0159] As mentioned above, when the fingerprints are the same, it is necessary to add a byte-by-byte comparison judgment to prevent the fingerprint algorithm from misjudging; and when the byte-by-byte comparison result is returned, a certain mechanism is also required to ensure that the object content corresponding to the fingerprint will not change during the byte-by-byte comparison.

[0160] When deduplicating a data object, if the fingerprint corresponding to the data object does not exist in the fingerprint library, it means that the data object must not be deleted; the fingerprint library generates the corresponding fingerprint and generates a unique identifier for the fingerprint.

[0161] When deduplicating a data object, if the fingerprint corresponding to the data object is found in the fingerprint library, a byte-by-byte comparison will be initiated to prevent misjudgment. When performing a byte-by-byte comparison, the unique identifier corresponding to the fingerprint is carried. When the byte-by-byte comparison returns the comparison result, the fingerprint library is checked and the following logical judgment is performed:

[0162] a) If the corresponding fingerprint cannot be found, it means that the corresponding fingerprint has been deleted. Add the fingerprint corresponding to the data object to the fingerprint library.

[0163] b) If the corresponding fingerprint can be found and the unique identifier of the fingerprint has not changed, it means that the content of the object corresponding to the fingerprint must not have changed during the byte-by-byte comparison. If the byte-by-byte comparison returns the same data, the reference count of the corresponding fingerprint is increased, and the data of the current data object is deleted. The two data objects reference the same data, achieving the effect of deduplication. If the byte-by-byte comparison returns different data, no deduplication is performed.

[0164] c) If the corresponding fingerprint can be found, but the unique identifier of the fingerprint has changed, it means that during the byte-by-byte comparison process, the object content corresponding to the fingerprint may have changed (the old fingerprint has been deleted, and the current fingerprint is the newly added fingerprint). In this case, you need to re-initiate the byte-by-byte comparison, carry the unique identifier of the newly added fingerprint, and wait for the return result.

[0165] In this embodiment, when it is determined that the fingerprints of two data objects are the same, a byte-by-byte comparison is also performed to accurately determine whether the two data are completely consistent, to prevent fingerprint algorithm conflicts and data errors; and when generating a fingerprint, a unique identifier is also generated for the fingerprint to prevent the data corresponding to the fingerprint from changing during the byte-by-byte comparison process, thereby preventing data errors.

[0166] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0167] Based on the same inventive concept, the embodiment of the present application also provides a data object deduplication processing device for implementing the data object deduplication processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data object deduplication processing device embodiments provided below can refer to the limitations of the data object deduplication processing method above, and will not be repeated here.

[0168] In an exemplary embodiment, as shown in FIG9 , a data object deduplication processing device 1 is provided, comprising: a candidate object acquisition module 10 and an object deduplication processing module 20, wherein:

[0169] A candidate object acquisition module 10, configured to respond to a deduplication processing request of a target data object and acquire a candidate data object having the same fingerprint as the target data object in the storage system;

[0170] The object deduplication processing module 20 is used to perform deduplication processing on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object if the target data object is the same as the candidate data object.

[0171] In one embodiment, the candidate object acquisition module 10 is further used for:

[0172] The target data object is fingerprinted to obtain a target fingerprint of the target data object; the target fingerprint is compared with multiple fingerprints in the fingerprint library; if there is a fingerprint identical to the target fingerprint in the fingerprint library, the fingerprint identical to the target fingerprint is obtained, and the data object corresponding to the fingerprint identical to the target fingerprint in the storage system is determined as a candidate data object.

[0173] In one embodiment, the candidate object acquisition module 10 is further used for:

[0174] If there is no fingerprint identical to the target fingerprint in the fingerprint database, the target fingerprint is added to the fingerprint database.

[0175] In one embodiment, the object deduplication processing module 20 is further used for:

[0176] When the byte-by-byte comparison between the target data object and the candidate data object is completed, obtain the reference unique identifier corresponding to the data object with the same fingerprint as the target data object in the current fingerprint library; based on the unique identifier and the reference unique identifier, update and verify the object content corresponding to the same fingerprint to obtain an update verification result; based on the update verification result, perform deduplication processing on the target data object.

[0177] In one embodiment, the object deduplication processing module 20 is further used for:

[0178] The unique identifier is compared with the reference unique identifier; if the unique identifier is consistent with the reference unique identifier, it is determined that the updated verification result is that the object content corresponding to the same fingerprint has not changed; if the unique identifier is inconsistent with the reference unique identifier, it is determined that the updated verification result is that the object content corresponding to the same fingerprint has changed.

[0179] In one embodiment, the object deduplication processing module 20 is further used for:

[0180] If the update verification result is that the object content corresponding to the same fingerprint has not changed, the target data object is deleted; if the update verification result is that the object content corresponding to the same fingerprint has changed, a byte-by-byte comparison operation is re-initiated on the target data object, and the target data object is deduplicated according to the comparison result.

[0181] In one of the embodiments, the data object deduplication processing device 1 further includes:

[0182] A byte-by-byte comparison module, used for comparing the target data object with the candidate data object byte-by-byte;

[0183] A first determination module, configured to determine that the target data object is identical to the candidate data object when each byte of the target data object is identical to each byte of the candidate data object;

[0184] The second determination module is used to determine that the target data object is different from the candidate data object when there is at least one byte difference between the target data object and the candidate data object.

[0185] Each module in the above-mentioned data object deduplication processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0186] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data object deduplication processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data object deduplication processing method is implemented.

[0187] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0189] In response to a deduplication processing request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system;

[0190] If the target data object is the same as the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0191] The implementation principles and technical effects of each step implemented by the processor in the embodiment of the present application are similar to the principles of the above-mentioned data object deduplication processing method, and will not be repeated here.

[0192] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0193] In response to a deduplication processing request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system;

[0194] If the target data object is the same as the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0195] The implementation principles and technical effects of the various steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned data object deduplication processing method, and will not be repeated here.

[0196] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0197] In response to a deduplication processing request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system;

[0198] If the target data object is the same as the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

[0199] The implementation principles and technical effects of the various steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned data object deduplication processing method, and will not be repeated here.

[0200] It should be noted that the personnel information (including but not limited to personnel equipment information, personnel personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the personnel or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0201] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0202] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data object deduplication processing method, characterized in that: The method comprises: In response to a deduplication request for a target data object, obtaining a candidate data object having the same fingerprint as the target data object in a storage system; If the target data object is the same as the candidate data object, deduplication processing is performed on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object.

2. The method according to claim 1, characterized in that The step of acquiring the candidate data object having the same fingerprint as the target data object in the storage system includes: Performing fingerprint calculation on the target data object to obtain a target fingerprint of the target data object; Comparing the target fingerprint with multiple fingerprints in a fingerprint library; In the case that the fingerprint database contains a fingerprint identical to the target fingerprint, the fingerprint identical to the target fingerprint is obtained, and a data object corresponding to the fingerprint identical to the target fingerprint in the storage system is determined as the candidate data object.

3. The method according to claim 2, characterized in that The method further comprises: When there is no fingerprint identical to the target fingerprint in the fingerprint library, the target fingerprint is added to the fingerprint library.

4. The method according to any one of claims 1 to 3, characterized in that: The deduplication processing of the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object includes: When the target data object and the candidate data object are compared byte by byte, obtaining a reference unique identifier corresponding to a data object with the same fingerprint as the target data object in the current fingerprint database; Performing update verification on the object content corresponding to the same fingerprint according to the unique identifier and the reference unique identifier to obtain an update verification result; Deduplication processing is performed on the target data object according to the update verification result.

5. The method according to claim 4, characterized in that The updating and verifying the object content corresponding to the same fingerprint according to the unique identifier and the reference unique identifier to obtain an updating and verification result includes: comparing the unique identifier with the reference unique identifier; If the unique identifier is consistent with the reference unique identifier, determining that the update verification result is that the object content corresponding to the same fingerprint has not changed; If the unique identifier is inconsistent with the reference unique identifier, it is determined that the update verification result is that the object content corresponding to the same fingerprint has changed.

6. The method according to claim 4, characterized in that The deduplication processing of the target data object according to the update verification result includes: If the update verification result is that the object content corresponding to the same fingerprint has not changed, deleting the target data object; If the update verification result is that the object content corresponding to the same fingerprint has changed, a byte-by-byte comparison operation is re-initiated for the target data object, and deduplication processing is performed on the target data object according to the comparison result.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Comparing the target data object with the candidate data object byte by byte; In a case where each byte of the target data object is identical to each byte of the candidate data object, determining that the target data object is identical to the candidate data object; In the case that there is at least one byte difference between the target data object and the candidate data object, it is determined that the target data object is different from the candidate data object.

8. A data object deduplication processing device, characterized in that: The device comprises: A candidate object acquisition module, configured to respond to a deduplication processing request of a target data object and acquire a candidate data object having the same fingerprint as the target data object in a storage system; The object deduplication processing module is used to perform deduplication processing on the target data object according to the unique identifier corresponding to the fingerprint of the candidate data object if the target data object is the same as the candidate data object.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images