Object deduplication method and apparatus

Abstract

Embodiments of the present application provide a method for deduplicating objects, which comprises: obtaining a first hash value of a first object and a first object identifier; determining whether the first object is a duplicate object according to the first hash value and the first object identifier; and performing a deduplication operation on the first object or a second object in a case where the first object is determined to be a duplicate object; wherein the second object comprises an object that is duplicated with the first object in a storage system. The technical solution of the embodiments of the present application determines whether there is a duplicate object through a hash value and an object identifier. In a case where there is a duplicate object, the duplicate object is cleaned to optimize the utilization of storage resources of the storage system and avoid waste of storage resources in the storage system. Meanwhile, since the storage of duplicate objects is reduced, the time and cost of data recovery in a case of data loss are also reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and in particular, to an object deduplication method and device, computer equipment and computer readable storage medium. BACKGROUND

[0002] With the development of computer technology, mass storage, read and write of various data have become the current hot issues. In data storage, object storage as an important data storage method, due to its high reliability, strong scalability, fast access speed and other advantages, has been more and more widely concerned and applied. However, with the continuous growth of data, a large amount of duplicate data will appear in object storage, which not only wastes space, but also increases management and maintenance costs.

[0003] It should be noted that the above content is not necessarily prior art, nor is it used to limit the patent protection scope of the present application. SUMMARY

[0004] Embodiments of the present application provide an object deduplication method and device, computer equipment and computer readable storage medium to solve or alleviate one or more technical problems proposed above.

[0005] One aspect of an embodiment of the present application provides an object deduplication method, which comprises:

[0006] obtaining a first hash value of a first object and a first object identifier;

[0007] determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier;

[0008] performing a deduplication operation on the first object or a second object in a case where it is determined that the first object belongs to a duplicate object; wherein the second object comprises an object duplicated with the first object in a storage system.

[0009] Optionally, the first object is a to-be-uploaded object.

[0010] Optionally, determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier comprises:

[0011] retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0012] determining that the first object does not belong to a duplicate object in a case where the second hash value is not retrieved;

[0013] wherein, in a case where it is determined that the first object does not belong to a duplicate object, the first object is physically stored in the storage system.

[0014] Optionally, determining whether the first object is a duplicate object according to the first hash value and the first object identifier comprises:

[0015] retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0016] in a case where the second hash value is retrieved and the first object identifier is identical to the second object identifier, determining that the first object is a duplicate object;

[0017] wherein, in a case where it is determined that the first object is a duplicate object, returning a message indicating that the response is successful.

[0018] Optionally, determining whether the first object is a duplicate object according to the first hash value and the first object identifier comprises:

[0019] retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0020] in a case where the second hash value is retrieved and the first object identifier is different from the second object identifier, physically storing the first object;

[0021] determining whether the first object and the second object are duplicates by comparing the first object and the second object.

[0022] Optionally, the method further comprises:

[0023] asynchronously establishing a first mapping relationship, the first mapping relationship representing a mapping relationship between the first object identifier and first metadata; wherein the first metadata comprises a physical storage address of the first object in the storage system;

[0024] asynchronously establishing a second mapping relationship, the second mapping relationship representing a mapping relationship between the first hash value and the first object identifier;

[0025] Correspondingly, determining whether the first object and the second object are duplicates by comparing the first object and the second object comprises:

[0026] in a case where the first hash value and the second hash value produce a hash collision and the first object identifier is different from the second object identifier, obtaining a first object associated with the first object identifier and a second object associated with the second object identifier; wherein the second hash value and the second object identifier are in a mapping relationship;

[0027] In a case that the first object and the second object are byte by byte identical, it is determined that the first object and the second object are duplicated.

[0028] Optionally, in a case that it is determined that the first object belongs to a duplicated object, a deduplication operation is performed on the first object or the second object, including:

[0029] In a case that it is determined that the first object and the second object are duplicated, according to the lifecycle information of the first object and the lifecycle information of the second object, the first object or the second object is selected as a recyclable object;

[0030] The recyclable object in the storage system is physically deleted; and

[0031] Information of first metadata of the first object and / or second metadata of the second object is updated.

[0032] Optionally, by comparing the first object and the second object, it is determined whether the first object and the second object are duplicated, further including:

[0033] In a case that the first object and the second object are not byte by byte identical, it is determined that the first object and the second object are not duplicated.

[0034] Optionally, the first object associated with the first object identifier and the second object associated with the second object identifier are obtained, including:

[0035] The first metadata of the first object is obtained through the first object identifier;

[0036] The second metadata of the second object is obtained through the second object identifier;

[0037] The first metadata is used for locating the first object in the storage system, the second metadata is used for locating the second object, and the second metadata includes a physical storage address of the second object in the storage system.

[0038] Optionally, the first object is a stored object.

[0039] Optionally, according to the first hash value and the first object identifier, it is determined whether the first object belongs to a duplicated object, including:

[0040] A second hash value identical to the first hash value is retrieved, and the second hash value maps a second object identifier;

[0041] In a case that the second hash value is not retrieved, it is determined that the first object does not belong to a duplicated object.

[0042] Optionally, determining whether the first object is a duplicate object according to the first hash value and the first object identifier comprises:

[0043] retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0044] in a case where the second hash value is retrieved and the first object identifier is different from the second object identifier, obtaining a first object associated with the first object identifier and a second object associated with the second object identifier;

[0045] determining whether the first object and the second object are duplicates by comparing the first object and the second object.

[0046] Optionally, in a case where it is determined that the first object is a duplicate object, performing a deduplication operation on the first object or the second object comprises:

[0047] in a case where it is determined that the first object and the second object are duplicates, selecting the first object or the second object as a recyclable object according to lifecycle information of the first object and lifecycle information of the second object;

[0048] physically deleting the recyclable object in the storage system; and

[0049] updating information of first metadata of the first object and / or second metadata of the second object.

[0050] Optionally, determining whether the first object and the second object are duplicates by comparing the first object and the second object further comprises:

[0051] in a case where the first object and the second object are not byte-by-byte identical, determining that the first object and the second object are not duplicates.

[0052] Another aspect of the embodiments of the present application provides an object deduplication device, the device comprising:

[0053] an obtaining module configured to obtain a first hash value of a first object and a first object identifier;

[0054] a determining module configured to determine whether the first object is a duplicate object according to the first hash value and the first object identifier;

[0055] a deduplication module configured to, in a case where it is determined that the first object is a duplicate object, perform a deduplication operation on the first object or a second object; wherein the second object comprises an object identical to the first object in a storage system.

[0056] Another aspect of the embodiments of the present application provides a computer device, comprising:

[0057] at least one processor; and

[0058] a memory in communication with the at least one processor;

[0059] wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0060] Another aspect of the embodiments of the present application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method as described above.

[0061] The embodiments of the present application can include the following advantages by using the above technical solutions: whether there is a duplicate object can be determined by the hash value and the object identifier. When there is a duplicate object, the object of the duplicate object is cleaned to optimize the storage resource utilization of the storage system as much as possible, and the waste of storage resources in the storage system is avoided as much as possible. At the same time, since the storage of the duplicate object is reduced, the time and cost of data recovery when data is lost are also reduced. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the application. In all the drawings, like reference numerals refer to like parts throughout the various figures. The key elements of the drawings are as follows:

[0063] Figure 1 A diagram of a running environment of an object deduplication method according to Embodiment One of the present application is schematically shown;

[0064] Figure 2 A flowchart of the object deduplication method according to Embodiment One of the present application is schematically shown;

[0065] Figure 3 A processing architecture in an exemplary application is schematically shown;

[0066] Figure 4 A processing flow in an exemplary application is schematically shown;

[0067] Figure 5 A block diagram of an object deduplication device according to Embodiment Two of the present application is schematically shown; and

[0068] Figure 6The hardware architecture schematic diagram of the computer device according to the embodiment three of the present application is shown schematically. DETAILED DESCRIPTION

[0069] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0070] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for the purpose of description, and should not be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features with "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that the technical solutions can be realized by those of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0071] In the description of the present application, it should be understood that the numerical reference numbers before the steps do not indicate the order of execution of the steps before and after, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as limiting the present application.

[0072] First, the explanations of the terms involved in the present application are provided:

[0073] Object storage: a computer data storage architecture that manages data as objects, each object typically includes data itself, a variable amount of metadata, and a globally unique identifier (Object ID).

[0074] Storage object: the most basic concept in cloud storage, refers to the file or data uploaded by the user in the cloud storage. Each object has a unique identifier (Object ID), and the object can be accessed and operated through the identifier.

[0075] Hash collision: two or more key values are calculated to the same index position by the hash function, thereby forming a collision.

[0076] Physical deletion: deleting data from storage media.

[0077] MD5: full name for Message-Digest Algorithm 5, is a commonly used hash function (Hash Function), used to compress any length of message into a 128-bit message digest (Message Digest). MD5 can be used in data integrity verification, password encryption and other fields.

[0078] Bucket: a container for storing objects. Each bucket has a unique name, and storage objects can be created, deleted and managed in the bucket. The bucket can be regarded as a top-level directory or container, which contains multiple objects.

[0079] ObjectName: the key name of the object, used to refer to a specific object in the bucket. The ObjectName in a bucket is unique.

[0080] ObjectKey: the unique identifier of the object in the object storage system, which is equal to Bucket / ObjectName.

[0081] Index: a data structure that can speed up data lookup. The index data node has the storage address of the actual file, because the index is constructed according to specific rules and algorithms, and when searching, following the rules of the index can quickly find the corresponding data node, thereby achieving the effect of quickly searching data.

[0082] Hash Index: is based on hash table implementation, and only the query of all columns of the exact match index is effective. For each row of data, the storage engine will calculate a hash code (Hash Code) for all index columns. The hash code is a smaller value, and the hash code calculated for different key values is different. Hash index stores all hash codes in the index, and at the same time saves a pointer to each data row in the hash table.

[0083] Secondly, in order to facilitate the understanding of the technical solution provided by the person skilled in the art, the related technology is described as follows:

[0084] Object storage technology is a distributed storage technology used to store and manage large-scale data objects. It provides a highly scalable and highly reliable data storage solution to meet the needs of application programs processing large-scale data. The existence of duplicate data in the object storage system will cause waste of storage space, and also increase the time and cost of data recovery. Specifically, the present inventors have found that: 1. For duplicate data uploaded, the object storage system has no explicit deduplication method, and duplicate data will apply for new storage space, causing resource waste; 2. For duplicate data that has been written into the storage system, there is no explicit method to eliminate duplicate data.

[0085] To this end, embodiments of the present application provide a technical solution to eliminate duplicate data in an object storage system. See the following for details.

[0086] Finally, for ease of understanding, an example operating environment is provided below.

[0087] As shown in Figure 1 , the environment schematic diagram includes a storage system 2, a network 4, and a client 6, wherein:

[0088] The storage system 2, as a storage platform, can be composed of multiple computing devices. The multiple computing devices can include virtualized computing instances. The virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices can load the virtual machines based on virtual images and / or other data defining specific software (e.g., operating systems, specialized applications, servers) for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device. The storage system 2 can include different storage functions, such as storing objects, storing metadata, etc.

[0089] In some embodiments, the storage system 2 is composed of multiple storage nodes, forming a distributed structure. The distributed storage system can employ various networking methods, wherein each storage node is used to provide computing and storage services. The number of storage nodes can be configured according to actual needs. The storage nodes can be disks or other non-volatile storage media. Taking disks as an example, the storage units can be single disks or disk arrays formed by multiple disks.

[0090] In some embodiments, the storage system 2 can have multiple storage forms, among which object storage is one of the storage forms. Object storage stores data as objects, each object can be bound with a unique identifier. This storage form can have very high scalability and availability, and supports large data sets. The distributed storage system can employ various architectures such as Amazon S3 (an object storage service provided by Amazon) for management.

[0091] The storage system 2 can provide storage, read, write, query, delete, and other services.

[0092] The storage system 2 can be configured to communicate with the client 6 and the like through the network 4.

[0093] The client 6 can be provided with a user access page for manipulating the storage system 2 or uploading objects, etc.

[0094] The following describes the technical solutions of the present application through multiple embodiments, with the storage system 2 as the execution subject. It should be noted that these embodiments can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein.

[0095] Embodiment One

[0096] Figure 2 The flowchart of the object deduplication method according to Embodiment One of the present application is schematically shown.

[0097] As Figure 2 shown, the object deduplication method can include steps S200-S204, in which:

[0098] Step S200 The first hash value and the first object identifier of the first object are obtained.

[0099] The first hash value: the file or data of the first object can be mapped to an output (first hash value) of a preset length through a hash function (Hash Function). The input and output can be any binary data.

[0100] The first object identifier: it can be the unique identifier (ObjectKey) of the first object, and the identifier in a Bucket is unique.

[0101] In this embodiment, the first object can be a stored object in the storage system 2, or a to-be-uploaded object.

[0102] In this embodiment, the first object can be a file, a picture, an audio, or other types of data.

[0103] Taking the first object as a file for example, the first hash value and the first identifier can be obtained in the following way: accessing the file and obtaining its file name as the first identifier. The MD5 algorithm is used to hash the file to obtain the first hash value. The first hash value and the first identifier are mapped and associated. In some embodiments, other hash algorithms can also be used to generate the first hash value, such as SHA-1, SHA-256, etc.

[0104] Step S202 According to the first hash value and the first object identifier, it is determined whether the first object belongs to a duplicate object.

[0105] The storage system 2 stores a large number of objects, and there can be a large number of duplicate objects in these objects, causing waste of storage resources.

[0106] The first object, whether a stored object or a to-be-uploaded object, should try to avoid the above duplicate storage behavior.

[0107] To this end, the first hash value of the first object and the hash values of other objects in the storage system 2 can be compared, and the comparison result can be used to preliminarily determine whether the first object is duplicated with other objects in the storage system 2. Further, the first object identifier can be used to further determine whether the first object is a duplicated object.

[0108] Specifically, a plurality of objects are stored in the storage system 2, each object has a unique object identifier and a hash value. If it is determined whether the first object is a duplicated object, the first hash value of the first object is obtained first, and then the first hash value is compared with the hash values of the plurality of objects (other objects) stored in the storage system 2 one by one. If they are all different, it is determined that the first object is not duplicated with other objects, i.e., the first object is not a duplicated object. If there is an object whose hash value is the same as the first hash value of the first object, then the object identifiers of the two objects are compared. If both of them are the same, it is determined that the first object is a duplicated object. If there is an object whose hash value is the same as the first hash value of the first object, but the object identifier of the object is different from the first object identifier of the first object, further determination is needed.

[0109] There are three cases according to the comparison results:

[0110] The first case: there is no hash value (of other objects) that is the same as the first hash value, and it is determined that the first object is not a duplicated object.

[0111] The second case: there is a hash value (of other objects) that is the same as the first hash value, and the object identifiers are the same, and it is determined that the first object is a duplicated object.

[0112] The third case: there is a hash value (of other objects) that is the same as the first hash value, but the object identifiers of the two objects are different, and further determination is needed.

[0113] Step S204 In the case where it is determined that the first object is a duplicated object, a deduplication operation is performed on the first object or the second object; wherein the second object includes an object duplicated with the first object in the storage system.

[0114] If the first object is an object to be uploaded to the storage system 2, the first object is rejected.

[0115] If the first object is an object already stored in the storage system 2, the first object or the second object is physically deleted, and one is left.

[0116] The object deduplication method provided by the embodiment can determine whether there is a duplicate object through the hash value and the object identifier. When there is a duplicate object, the object of the duplicate object is cleaned to optimize the storage resource utilization of the storage system 2 as much as possible, and waste of storage resources in the storage system 2 is avoided as much as possible. At the same time, since the storage of the duplicate object is reduced, the time and cost of data recovery when data is lost are also reduced.

[0117] As described above, the first object can be a stored object in the storage system 2 or a to-be-uploaded object. Different object types can correspond to different coping strategies. The operations of the two object types are further introduced below respectively.

[0118] In an optional embodiment, the first object is a to-be-uploaded object. When the first object is a duplicate object, the storage of the first object can be rejected, so that a new storage space does not need to be applied, and the waste of bandwidth and storage resources is reduced.

[0119] As a to-be-uploaded object, the first object can encounter various situations:

[0120] Case (I): there is no hash value identical to the first hash value of the first object in the storage system 2;

[0121] Case (II): there is a hash value (second hash value) identical to the first hash value in the storage system 2;

[0122] Case (II) can include case (1) and case (2):

[0123] Case (1): the first object identifier corresponding to the first hash value is identical to the second object identifier corresponding to the second hash value;

[0124] Case (2): the first object identifier corresponding to the first hash value is not identical to the second object identifier corresponding to the second hash value.

[0125] The various cases will be introduced below.

[0126] Case (I)

[0127] In an optional embodiment, the step S202 of “determining whether the first object is a duplicate object according to the first hash value and the first object identifier” can include: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case where the second hash value is not retrieved, it is determined that the first object is not a duplicate object; wherein, in the case where it is determined that the first object is not a duplicate object, the first object is physically stored in the storage system.

[0128] The hash function has the characteristics of collision resistance and one-way.

[0129] Different objects, even with slight differences, will produce different hash values. In the case where the second hash value is not retrieved, it means that there is no object in the storage system 2 that is duplicated with the first object, so the uploading of the first object is accepted and stored. In this optional embodiment, it can be effectively guaranteed that the uploaded object is new data.

[0130] Case (1) in case (two)

[0131] In an optional embodiment, step S202 "determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier" can include: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case where the second hash value is retrieved and the first object identifier and the second object identifier are identical, determining that the first object belongs to a duplicate object; wherein in the case where it is determined that the first object belongs to a duplicate object, the first object is rejected and a message indicating a successful response is returned.

[0132] Taking the hash value MD5 as an example, the probability of MD5 hash collision is 1 / (2^128), and the same object identifier means that it is the same uploader. The case of MD5 hash collision of the object to be uploaded by the same uploader can be ignored.

[0133] In the above embodiment, the hash values and object identifiers of the first object and the second object are identical, indicating that the first object and the second object are duplicate objects. Therefore, the storage system 2 can reject the storage of the first object, reducing the waste of bandwidth and storage resources. Since there is a second object identical to the first object in the storage system 2, although the first object is rejected, the uploading request of the first object is successfully responded, so a message indicating a successful response is returned.

[0134] Case (2) in case (two)

[0135] In an optional embodiment, step S202 "determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier" can include: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case where the second hash value is retrieved and the first object identifier and the second object identifier are different, physically storing the first object; and determining whether the first object and the second object are duplicated by comparing the first object and the second object.

[0136] In some cases, the hash collision does not represent the data is completely consistent. Therefore, in the case that the hash values of the first object and the second object are the same but the object identifiers are different, the uploading of the first object is accepted and stored first, ensuring the uploading efficiency. Then, after the first object is stored in the storage system 2, it is further compared whether the first object and the second object are the same, and according to the comparison result, it is determined whether to clean up the object.

[0137] In order not to affect the uploading of the first object, the comparison of the first object and the second object can be implemented asynchronously.

[0138] In an optional embodiment, after the first object is physically stored, the method can further include:

[0139] Asynchronously establishing a first mapping relationship, the first mapping relationship representing a mapping relationship between the first object identifier and first metadata; wherein the first metadata includes a physical storage address of the first object in the storage system;

[0140] Asynchronously establishing a second mapping relationship, the second mapping relationship representing a mapping relationship between the first hash value and the first object identifier; it should be noted that the second mapping relationship can also be referred to as a hash index;

[0141] Correspondingly, by comparing the first object and the second object, it is determined whether the first object and the second object are duplicated, including:

[0142] In the case that the first hash value and the second hash value produce hash collision, and the first object identifier and the second object identifier are not the same, the first object associated with the first object identifier and the second object associated with the second object identifier are obtained; wherein the second hash value and the second object identifier are in a mapping relationship;

[0143] In the case that the first object and the second object are byte-by-byte identical, it is determined that the first object and the second object are duplicated.

[0144] For example, a large number of objects are stored in the storage system 2. Each object is associated with metadata. The metadata of the object can include various information of the object, such as name, size, owner, creation time, expiration time, and physical storage address. In order to facilitate management and improve management efficiency, two layers of mapping relationships are stored. The two layers of mapping relationships are as follows:

[0145] (1) The first layer of mapping relationship between the object identifier of the object and the metadata of the object;

[0146] (2) The second layer of mapping relationship (hash index) between the object identifier of the object and the hash value of the object.

[0147] The relationship data of the two-layer mapping relationship can be stored in the metadata server or stored elsewhere.

[0148] Therefore, after the first object is physically stored, the two-layer mapping relationship of the first object is also established and stored. Then, whether the first object is a duplicate object is determined based on the content of the two-layer mapping relationship (the first mapping relationship and the second mapping relationship). Specifically, if the first hash value of the first object is the same as the hash value of an object in the storage system 2 (the second hash value of the second object), but the first object identifier of the first object is different from the second object identifier of the second object, the first object and the second object are further compared to determine whether they are the same, and whether the cleaning of the object is performed according to the comparison result. If the first object and the second object are the same, the first object or the second object is physically deleted. It should be noted that the establishment of the mapping relationship of the first object, the determination of whether the first object is a duplicate object, and the deletion are all performed asynchronously. In this way, the uploading efficiency is ensured, and the duplicate objects can also be effectively removed.

[0149] In an optional embodiment, the step S204 "performing a deduplication operation on the first object or the second object in a case where it is determined that the first object is a duplicate object" comprises:

[0150] In a case where it is determined that the first object and the second object are duplicates, selecting the first object or the second object as a recyclable object according to the life cycle information of the first object and the life cycle information of the second object;

[0151] Physically deleting the recyclable object in the storage system; and

[0152] Performing information update on the first metadata of the first object and / or the second metadata of the second object.

[0153] The whole process of an object from creation to deletion is called the life cycle of the object. In order to save space, the object whose life cycle ends (expired) needs to be deleted. When two objects are duplicates, the object whose life cycle end time is earlier can be deleted as a recyclable object. When two objects are duplicates and the life cycle end times are the same, the object whose uploading time is later is selected as a recyclable object. In addition, the metadata of the recyclable object is updated. For example, in a case where object A and object B are duplicates and object A is physically deleted, the metadata of object A can be updated, such as updating the physical storage address in the metadata of object A to the physical storage address of object B, so that object B can be accessed through the metadata of object A. In this embodiment, the deletion of the duplicate object is determined based on the life cycle information, and the deletion of the duplicate data is further optimized.

[0154] In an optional embodiment, the determining whether the first object and the second object are duplicated by comparing the first object and the second object further comprises:

[0155] In the case that the first object and the second object are not byte by byte identical, it is determined that the first object and the second object are not duplicated.

[0156] In the optional embodiment, the first object and the second object are not byte by byte identical, which means that the first object and the second object are not duplicated objects. Therefore, the first object can be stored in the storage system 2 to realize object storage.

[0157] In the storage system 2, different objects correspond to different physical storage addresses.

[0158] The obtaining the first object associated with the first object identifier and the second object associated with the second object identifier can comprise:

[0159] The first metadata of the first object is obtained through the first object identifier.

[0160] The second metadata of the second object is obtained through the second object identifier.

[0161] The first metadata is used to locate the first object in the storage system, and the second metadata is used to locate the second object. The second metadata comprises a physical storage address of the second object in the storage system.

[0162] In the case that the hash values are the same but the object identifiers are different, the respective physical storage addresses are found through the respective object identifiers, and then the first object and the second object in the storage system 2 are read through the respective physical storage addresses. This way of first doing hash collision through hash index and then querying and reading the object based on the object identifier can efficiently determine the duplicated object and then clean up the duplicated object, thereby saving the storage resources caused by the duplicated data.

[0163] In an optional embodiment, the first object is a stored object. When the first object is a duplicated object, a deduplication operation is performed to alleviate the storage resource cost caused by the duplicated data in the storage system 2 and reduce the time and cost of data recovery.

[0164] As a stored object, the first object can encounter various situations:

[0165] Situation (I): there is no hash value identical to the first hash value of the first object in the storage system 2.

[0166] Situation (II): there is a hash value (second hash value) identical to the first hash value in the storage system 2.

[0167] Case (2) can include case (1) and case (2):

[0168] Case (1): the first object identifier corresponding to the first hash value is the same as the second object identifier corresponding to the second hash value;

[0169] Case (2): the first object identifier corresponding to the first hash value is not the same as the second object identifier corresponding to the second hash value.

[0170] The various cases will be introduced below.

[0171] Case (1)

[0172] In an optional embodiment, the step S202 “determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier” can include: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case where the second hash value is not retrieved, determining that the first object does not belong to a duplicate object. Different objects will produce different hash values even if the difference is small. In the case where the second hash value is not retrieved, it indicates that there is no object identical to the first object in the storage system 2. In this embodiment, the first object can be efficiently determined not to belong to a duplicate object through hash collision.

[0173] Case (1) in case (2)

[0174] In an optional embodiment, the step S202 “determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier” can include: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case where the second hash value is retrieved, and the first object identifier is the same as the second object identifier, determining that the first object belongs to a duplicate object.

[0175] In the above embodiments, the hash values and object identifiers of the first object and the second object are the same, which indicates that the first object and the second object are duplicate objects. Therefore, the storage system 2 can perform a deduplication operation to reduce the waste of storage resources.

[0176] Case (2) in case (2)

[0177] In an optional embodiment, the step S202 "determining whether the first object belongs to the duplicate object according to the first hash value and the first object identifier" can comprise: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in the case that the second hash value is retrieved and the first object identifier is different from the second object identifier, obtaining the first object associated with the first object identifier and a second object associated with the second object identifier; and determining whether the first object and the second object are duplicate by comparing the first object and the second object. In this embodiment, in the case that the hash values of the first object and the second object are identical but the object identifiers are different, the first object and the second object are further compared to determine whether they are identical, and the comparison result is used to determine whether to clean up the objects.

[0178] In an optional embodiment, the step S204 "performing the deduplication operation on the first object or the second object in the case that the first object is determined to belong to the duplicate object" comprises:

[0179] In the case that the first object and the second object are determined to be duplicate, the first object or the second object is selected as the recyclable object according to the lifecycle information of the first object and the lifecycle information of the second object;

[0180] physically deleting the recyclable object in the storage system; and

[0181] updating the first metadata of the first object and / or the second metadata of the second object.

[0182] The whole process of an object from creation to deletion is called the lifecycle of the object. In order to save space, the object whose lifecycle ends (expired) needs to be deleted. When two objects are duplicate, the object whose lifecycle end time is earlier can be deleted as the recyclable object. When two objects are duplicate and the lifecycle end times are identical, the object whose uploading time is later is selected as the recyclable object. In addition, the metadata of the recyclable object is updated. For example, in the case that object A and object B are duplicate and object A is physically deleted, the metadata of object A can be updated, such as updating the physical storage address in the metadata of object A to the physical storage address of object B, so that object B can be accessed through the metadata of object A. In this embodiment, the deletion of the duplicate object is determined based on the lifecycle information, and the deletion of the duplicate data is further optimized.

[0183] In an optional embodiment, the step of determining whether the first object and the second object are duplicate by comparing the first object and the second object further comprises:

[0184] In the case that the first object and the second object are not byte by byte identical, it is determined that the first object and the second object are not duplicated.

[0185] In the optional embodiment, the first object and the second object are not byte by byte identical, which means that the first object and the second object are not duplicated objects. Therefore, the first object can be continuously stored in the storage system 2 to realize object storage.

[0186] In order to make the present application easier to understand, the following will be described in combination with an example application. Figure 3 and Figure 4 An example application is provided.

[0187] In the example application, for data storage, two layers of mapping relationships can be set, one layer of mapping relationship is (bucket / objectName, meta), and the other layer of mapping relationship is (MD5, bucket / objectName).

[0188] When uploading the object object_1 metadata meta_1 through the S3 gateway, if the object_1 metadata already exists (judged according to bucket / objectName) and the MD5 values are the same, the metadata other information is updated and directly returned.

[0189] In the case that the MD5 values are the same but the object identifiers (bucket / objectName) are different, the object_1 is uploaded (S400) and the following operations are performed:

[0190] S402: The computing node reads the MD5 value of the object_1.

[0191] S404: The computing node creates a hash index (i.e., the second layer of mapping relationship) asynchronously with the MD5 value of the object_1 as the key and the ObjectKey of the object_1 as the value.

[0192] S406: The computing node judges whether a hash collision occurs.

[0193] If no hash collision occurs, the asynchronous process ends.

[0194] If the hash index already exists, it means that there is an object_2 with the same MD5 value as the object_1 in the storage system, and S408 is entered.

[0195] S408: The computing node reads the object_2_name pointed by the object_2 MD5 in the collision object object_2 according to the established hash index.

[0196] S410: The computing node obtains object_2 metadata meta_2 according to object_2_name.

[0197] S412: The computing node sends the metadata [meta_1, meta_2] of the two objects to the message queue in a producer role.

[0198] S414: The offline task reads the above message in the message queue in a consumer role.

[0199] S416: The offline task reads the data of the two objects from the storage node.

[0200] S418: The offline task performs a byte-by-byte comparison on the data of the two objects.

[0201] S420: The offline task determines whether the data of the two objects are completely identical.

[0202] If no, the asynchronous process ends; if yes, go to S422.

[0203] S422: The offline task determines (filters) the recyclable object according to the expiration time and the creation time of the two objects.

[0204] The filtering logic is: if there is an expiration time, select the object with the earlier expiration time; if not, select the object with the later upload time.

[0205] S424: The offline task updates the physical storage location information of the recyclable object metadata, pointing to the physical storage location of the object collided therewith;

[0206] S426: The storage node releases the physical storage resource of the recyclable object.

[0207] Embodiment Two

[0208] Figure 5 A block diagram of an object deduplication device according to Embodiment Two of the present application is shown schematically, which can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of completing a specific function, and the functions of each program module in the embodiments will be described in detail below. As shown in the figure, the object deduplication device 500 can include an obtaining module 510, a determining module 520, and a deduplication module 530, wherein: Figure 5

[0209] The obtaining module 510 is configured to obtain a first hash value of a first object and a first object identifier.

[0210] ​determining module 520, configured to determine whether the first object is a duplicate object according to the first hash value and the first object identifier;

[0211] de-duplication module 530, configured to perform a de-duplication operation on the first object or a second object in a case where the first object is determined to be a duplicate object; the second object includes an object duplicated with the first object in the storage system.

[0212] In an optional embodiment, the first object is an object to be uploaded.

[0213] In an optional embodiment, the determining module 520 is further configured to:

[0214] retrieve a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0215] determine that the first object is not a duplicate object in a case where the second hash value is not retrieved;

[0216] In a case where it is determined that the first object is not a duplicate object, the first object is physically stored in the storage system.

[0217] In an optional embodiment, the determining module 520 is further configured to:

[0218] retrieve a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0219] determine that the first object is a duplicate object in a case where the second hash value is retrieved and the first object identifier is identical to the second object identifier;

[0220] In a case where it is determined that the first object is a duplicate object, a message indicating a successful response is returned.

[0221] In an optional embodiment, the determining module 520 is further configured to:

[0222] retrieve a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0223] physically store the first object in a case where the second hash value is retrieved and the first object identifier is different from the second object identifier.

[0224] determine whether the first object and the second object are duplicated by comparing the first object and the second object.

[0225] In an optional embodiment, the apparatus further includes an asynchronous module configured to:

[0226] asynchronously establishing a first mapping relationship, the first mapping relationship representing a mapping relationship between the first object identifier and first metadata; wherein the first metadata comprises a physical storage address of the first object in the storage system;

[0227] asynchronously establishing a second mapping relationship, the second mapping relationship representing a mapping relationship between the first hash value and the first object identifier;

[0228] Correspondingly, the determining module 520 is further configured to:

[0229] in a case where the first hash value and the second hash value cause a hash collision, and the first object identifier and the second object identifier are not identical, obtaining a first object associated with the first object identifier and a second object associated with the second object identifier; wherein the second hash value and the second object identifier are in a mapping relationship;

[0230] in a case where the first object and the second object are identical byte by byte, determining that the first object and the second object are duplicated.

[0231] In an optional embodiment, the determining module 520 is further configured to:

[0232] in a case where it is determined that the first object and the second object are duplicated, selecting the first object or the second object as a recyclable object according to lifecycle information of the first object and lifecycle information of the second object;

[0233] physically deleting the recyclable object in the storage system; and

[0234] performing information updating on first metadata of the first object and / or second metadata of the second object.

[0235] In an optional embodiment, the determining module 520 is further configured to:

[0236] in a case where the first object and the second object are not identical byte by byte, determining that the first object and the second object are not duplicated.

[0237] In an optional embodiment, the determining module 520 is further configured to:

[0238] obtaining first metadata of the first object through the first object identifier;

[0239] obtaining second metadata of the second object through the second object identifier;

[0240] The first metadata is used to locate the first object in the storage system, and the second metadata is used to locate the second object, and the second metadata comprises a physical storage address of the second object in the storage system.

[0241] In an optional embodiment, the first object is a stored object.

[0242] In an optional embodiment, the determining module 520 is further configured to:

[0243] retrieve a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0244] In a case where the second hash value is not retrieved, determine that the first object does not belong to a duplicate object.

[0245] In an optional embodiment, the determining module 520 is further configured to:

[0246] retrieve a second hash value identical to the first hash value, the second hash value mapping a second object identifier;

[0247] In a case where the second hash value is retrieved and the first object identifier is different from the second object identifier, obtain a first object associated with the first object identifier and a second object associated with the second object identifier;

[0248] determine whether the first object and the second object are duplicates by comparing the first object and the second object.

[0249] In an optional embodiment, the deduplication module 530 is further configured to:

[0250] In a case where the first object and the second object are determined to be duplicates, select the first object or the second object as a recyclable object according to lifecycle information of the first object and lifecycle information of the second object;

[0251] physically delete the recyclable object in the storage system; and

[0252] update information of the first metadata of the first object and / or the second metadata of the second object.

[0253] In an optional embodiment, the determining module 520 is further configured to:

[0254] In a case where the first object and the second object are not byte-by-byte identical, determine that the first object and the second object are not duplicates.

[0255] Embodiment Three

[0256] Figure 6 A schematic diagram of a hardware architecture of a computer device 10000 suitable for implementing the object deduplication method according to Embodiment Three of the present application is shown. In some embodiments, the computer device 10000 can be a rack server, a blade server, a tower server, or a cabinet server (including a standalone server, or a server cluster composed of multiple servers), etc. As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030, which are communicatively linked via a system bus. Figure 6

[0257] The memory 10010 includes at least one type of computer-readable storage media, which includes a flash memory, a hard disk, a multimedia card micro, a card-type memory (e.g., SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 10010 can include both an internal storage module and an external storage device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of the object deduplication method, and the like. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0258] The processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication of the computer device 10000, and the like. In this embodiment, the processor 10020 is used to run program codes or process data stored in the memory 10010.

[0259] ​The network interface 10030 can include a wireless network interface or a wired network interface, and is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, and the like wireless or wired network.

[0260] It should be noted that, Figure 6 Only the computer device with components 10010-10030 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0261] In this embodiment, the object deduplication method stored in the memory 10010 can also be divided into one or more program modules, and executed by one or more processors (such as the processor 10020) to complete the embodiments of the present application.

[0262] Embodiment Four

[0263] The embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium has a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the object deduplication method in the embodiments.

[0264] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device, such as program codes of the object deduplication method in the embodiments, etc. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0265] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computer device, which can be concentrated on a single computer device or distributed on a network composed of multiple computer devices, and optionally, each module or each step can be realized by program codes executable by a computer device, so that each module or each step can be stored in a storage device and executed by a computer device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0266] It should be noted that the above is only the preferred embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of deduplicating objects, the method comprising: The method comprises: obtaining a first hash value of a first object and a first object identifier; determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier; in a case where it is determined that the first object belongs to a duplicate object, performing a deduplication operation on the first object or a second object; wherein the second object comprises an object duplicated with the first object in a storage system; wherein the first object is a to-be-uploaded object; determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier comprises: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in a case where the second hash value is retrieved and the first object identifier and the second object identifier are different, physically storing the first object; determining whether the first object and the second object are duplicated by comparing the first object and the second object; the method further comprises: asynchronously establishing a first mapping relationship, the first mapping relationship representing a mapping relationship between the first object identifier and first metadata; wherein the first metadata comprises a physical storage address of the first object in the storage system; asynchronously establishing a second mapping relationship, the second mapping relationship representing a mapping relationship between the first hash value and the first object identifier; correspondingly, determining whether the first object and the second object are duplicated by comparing the first object and the second object comprises: in a case where the first hash value and the second hash value produce a hash collision and the first object identifier and the second object identifier are not identical, obtaining a first object associated with the first object identifier and a second object associated with the second object identifier; wherein the second hash value and the second object identifier are in a mapping relationship; in a case where the first object and the second object are identical byte by byte, determining that the first object and the second object are duplicated.

2. The method of claim 1, wherein, determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier comprises: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in a case where the second hash value is not retrieved, determining that the first object does not belong to a duplicate object; wherein, in a case where it is determined that the first object does not belong to a duplicate object, the first object is physically stored in the storage system.

3. The method of claim 1, wherein, determining whether the first object belongs to a duplicate object according to the first hash value and the first object identifier comprises: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in a case where the second hash value is retrieved and the first object identifier and the second object identifier are identical, determining that the first object belongs to a duplicate object; wherein, in a case where it is determined that the first object belongs to a duplicate object, a message representing a successful response is returned.

4. The method of claim 1, wherein, in a case where it is determined that the first object belongs to a duplicate object, performing a deduplication operation on the first object or a second object comprises: determining, in a case where it is determined that the first object and the second object are duplicated, whether the first object or the second object is a recoverable object according to lifecycle information of the first object and lifecycle information of the second object; physically deleting the recoverable object in the storage system; and updating information of first metadata of the first object and / or second metadata of the second object.

5. The method of claim 1, wherein, The determining whether the first object and the second object are duplicated by comparing the first object and the second object further includes: determining that the first object and the second object are not duplicated in a case where the first object and the second object are not byte-by-byte identical.

6. The method according to any one of claims 1, 4 or 5, characterized in that, The obtaining the first object associated with the first object identifier and the second object associated with the second object identifier includes: obtaining first metadata of the first object through the first object identifier; obtaining second metadata of the second object through the second object identifier; The first metadata is used for locating the first object in the storage system, and the second metadata is used for locating the second object, and the second metadata includes a physical storage address of the second object in the storage system.

7. The method of claim 1, wherein, The first object is a stored object.

8. The method of claim 7, wherein, The determining whether the first object is a duplicated object according to the first hash value and the first object identifier includes: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; determining that the first object is not a duplicated object in a case where the second hash value is not retrieved.

9. The method of claim 7, wherein, The determining whether the first object is a duplicated object according to the first hash value and the first object identifier includes: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; in a case where the second hash value is retrieved and the first object identifier and the second object identifier are different, obtaining the first object associated with the first object identifier and the second object associated with the second object identifier; The determining whether the first object and the second object are duplicated by comparing the first object and the second object.

10. The method of claim 9, wherein, The performing a deduplication operation on the first object or the second object in a case where it is determined that the first object is a duplicated object includes: determining, in a case where it is determined that the first object and the second object are duplicated, whether the first object or the second object is a recoverable object according to lifecycle information of the first object and lifecycle information of the second object; physically deleting the recoverable object in the storage system; and updating information of first metadata of the first object and / or second metadata of the second object.

11. The method of claim 9, wherein, The determining whether the first object and the second object are duplicated by comparing the first object and the second object further includes: determining that the first object and the second object are not duplicated in a case where the first object and the second object are not byte-by-byte identical.

12. An object deduplication apparatus, characterized by comprising: The apparatus includes: an obtaining module, configured to obtain a first hash value of a first object and a first object identifier; determining whether the first object is a duplicate object according to the first hash value and the first object identifier; de-duplication module, configured to perform a de-duplication operation on the first object or a second object if it is determined that the first object is a duplicate object; wherein the second object comprises an object duplicated with the first object in the storage system; wherein the first object is a to-be-uploaded object; determining whether the first object is a duplicate object according to the first hash value and the first object identifier comprises: retrieving a second hash value identical to the first hash value, the second hash value mapping a second object identifier; physically storing the first object if the second hash value is retrieved and the first object identifier is different from the second object identifier; determining whether the first object and the second object are duplicated by comparing the first object and the second object; further comprising: asynchronously establishing a first mapping relationship, the first mapping relationship representing a mapping relationship between the first object identifier and first metadata; wherein the first metadata comprises a physical storage address of the first object in the storage system; asynchronously establishing a second mapping relationship, the second mapping relationship representing a mapping relationship between the first hash value and the first object identifier; correspondingly, determining whether the first object and the second object are duplicated by comparing the first object and the second object comprises: in a case where the first hash value and the second hash value produce a hash collision and the first object identifier is different from the second object identifier, obtaining a first object associated with the first object identifier and a second object associated with the second object identifier; wherein the second hash value and the second object identifier are in a mapping relationship; in a case where the first object and the second object are identical byte by byte, determining that the first object and the second object are duplicated.

13. A computer device, characterized by comprise: at least one processor; and a memory in communication connection with the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method of any one of claims 1 to 11.

15. A computer program product comprising computer instructions, characterised in that, The computer instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 11.

Images