Hash-based key-to-blob conversion method and system

Abstract

Described is a method comprising: computing a first index based on a hash of a first key for a first object to be stored in a memory device; determining availability of a first segment on the memory device corresponding to the first index; in response to determining that the first segment corresponding to the first index is not available for storage, computing a second index based on the hash of the first key for the first object; determining availability of a second segment on the memory device corresponding to the second index; and adding an indicator of a location of the second segment to a conflict table in first metadata of the first segment.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 139,655, filed January 20, 2021, and U.S. Provisional Patent Application No. 17 / 321,326, filed May 14, 2021, which are incorporated herein by reference in their entirety. Technical Field

[0003] This application relates to key-value systems, and more specifically, to efficient key-value to block conversion methods and systems based on hashing. Background Technology

[0004] Some storage devices cannot communicate with applications (e.g., software applications) without relying on converters to convert key-value pairs to blocks and vice versa. Conventional techniques are resource-intensive and have high overhead. Furthermore, some techniques do not work well when scaling. Therefore, more efficient conversion techniques are needed. Summary of the Invention

[0005] According to an embodiment, a method is described. The method may include: calculating a first index based on a hash of a first key for a first object to be stored in a memory device; determining the availability of a first segment on the memory device corresponding to the first index; in response to determining that the first segment corresponding to the first index is unavailable for storage, calculating a second index based on a hash of the first key for the first object; determining the availability of a second segment on the memory device corresponding to the second index; and adding an indicator of the location of the second segment to a conflict table in the first metadata of the first segment.

[0006] The method may also include: writing the first object into the second segment in response to determining that the second segment is available.

[0007] Calculating the second index may include extracting bits from each byte of the hash to form the second index.

[0008] Determining the availability of the first segment includes: reading a status identifier from the first metadata, wherein the status identifier indicates a state corresponding to the first segment being unavailable for storage.

[0009] The method may further include: in response to the collision table being empty, writing the association between the hash of the first key and the indicator of the position of the second segment into the collision table.

[0010] The calculation of the first index involves using the Secure Hash Algorithm 256 (SHA-256).

[0011] The method may further include: receiving a get request indicating a first key; calculating a first index based on the hash of the first key; reading first metadata of a first segment corresponding to the first index; determining, in response to reading the first metadata, that the hash of the first key is inconsistent with a stored hash stored in the first metadata; reading a collision table in the first metadata to determine that the first object is stored in a second segment; and retrieving the first object from the second segment in response to reading the collision table.

[0012] The method may further include: receiving a delete request identifying a first key; calculating a first index based on the hash of the first key; reading first metadata of a first segment corresponding to the first index; determining, in response to reading the first metadata, that the hash of the first key is inconsistent with a stored hash stored in the first metadata; reading a conflict table in the first metadata to determine that the first object is stored in a second segment; and deleting the first object from the second segment in response to reading the conflict table.

[0013] The method may further include: receiving a deletion request identifying a first key; calculating a first index based on the hash of the first key; reading first metadata of a first segment corresponding to the first index; in response to reading the first metadata, determining that the hash of the first key matches a stored hash stored in the first metadata; determining that the collision table in the first metadata includes an indicator of the location of the second segment; setting the size of the collision table to zero; and deleting the first object from the second segment.

[0014] The method may further include: receiving a deletion request identifying a first key; calculating a first index based on the hash of the first key; reading first metadata of a first segment corresponding to the first index; determining, in response to reading the first metadata, that the hash of the first key matches a stored hash stored in the first metadata; determining, in response to determining that the hash of the first key matches a stored hash stored in the first metadata, that the conflict table in the first metadata is empty; and deleting the first object from a second segment in response to reading the conflict table.

[0015] According to another embodiment, an object storage system configured to store objects is described. The object storage system may include a segmented memory device configured to: compute a first index based on a hash of a first key for a first object to be stored in the memory device; determine the availability of a first segment on the memory device corresponding to the first index; in response to determining that the first segment corresponding to the first index is unavailable for storage, compute a second index based on a hash of the first key for the first object; determine the availability of a second segment on the memory device corresponding to the second index; and add an indicator of the location of the second segment to a conflict table in the first metadata of the first segment.

[0016] The memory device can be configured to write a first object into a second segment in response to determining that the second segment is available.

[0017] Calculating the second index involves extracting bits from each byte of the hash to form the second index.

[0018] Determining the availability of the first segment includes: reading a status identifier from the first metadata, wherein the status identifier indicates a state corresponding to the first segment being unavailable for storage.

[0019] The memory device can also be configured to write the association between the hash of the first key and the indicator of the location of the second segment into the collision table in response to the collision table being empty.

[0020] The calculation of the first index involves using the Secure Hash Algorithm 256 (SHA-256).

[0021] The memory device may also be configured to: receive a retrieval request indicating a first key; calculate a first index based on the hash of the first key; read first metadata of a first segment corresponding to the first index; determine, in response to reading the first metadata, that the hash of the first key is inconsistent with a stored hash stored in the first metadata; read a collision table in the first metadata to determine that the first object is stored in a second segment; and, in response to reading the collision table, retrieve the first object from the second segment.

[0022] The memory device can also be configured to: receive a deletion request identifying a first key; calculate a first index based on the hash of the first key; read first metadata of a first segment corresponding to the first index; determine, in response to reading the first metadata, that the hash of the first key is inconsistent with a stored hash stored in the first metadata; read a collision table in the first metadata to determine that the first object is stored in a second segment; and delete the first object from the second segment in response to reading the collision table.

[0023] The memory device can also be configured to: receive a deletion request identifying a first key; calculate a first index based on the hash of the first key; read first metadata of a first segment corresponding to the first index; in response to reading the first metadata, determine that the hash of the first key matches a stored hash stored in the first metadata; determine that the collision table in the first metadata includes an indicator of the location of the second segment; set the size of the collision table to zero; and delete the first object from the second segment.

[0024] The memory device is further configured to: receive a deletion request identifying a first key; calculate a first index based on the hash of the first key; read first metadata of a first segment corresponding to the first index; in response to reading the first metadata, determine that the hash of the first key matches a stored hash stored in the first metadata; in response to determining that the hash of the first key matches a stored hash stored in the first metadata, determine that the collision table in the first metadata is empty; and in response to reading the collision table, delete the first object from a second segment.

[0025] According to another embodiment, a method is described, comprising: receiving a retrieval request indicating a first key; calculating a first index based on a hash of the first key for an object to be retrieved from a memory device; reading first metadata of a first segment corresponding to the first index; determining, in response to reading the metadata, that the hash of the first key is inconsistent with a stored hash stored in the first metadata; reading a collision table in the first metadata to determine that the object is stored in a second segment; and retrieving the object from the second segment in response to reading the collision table.

[0026] The collision table may include a second index based on a hash of the first key and an offset of an object stored in the first segment, wherein the offset is an association between the hash of the first key and an indicator of the location of the second segment.

[0027] The scope of this invention is defined by the claims, which are incorporated herein by reference. Those skilled in the art will be able to more fully understand embodiments of the invention, and realize its additional advantages, by considering the following detailed description of one or more embodiments. Reference will be made to the accompanying drawings, which will first be briefly described. Attached Figure Description

[0028] Figure 1 This is a block diagram illustrating a memory device for key-value storage using hash-based block translation for communication with an application, according to an embodiment.

[0029] Figure 2 This is a flowchart of a method for finding an available slot on a memory device according to an embodiment.

[0030] Figure 3 This is a flowchart of a method for placing an object onto an available slot on a memory device, according to an embodiment.

[0031] Figure 4 This is a flowchart of a method for retrieving an object from a memory device according to an embodiment.

[0032] Figure 5 This is a flowchart of a method for deleting objects on a memory device according to an embodiment.

[0033] Figure 6 This is a flowchart of methods for recovering from a system crash or reboot.

[0034] Figure 7 Example metadata is shown according to an embodiment.

[0035] The embodiments and advantages of this disclosure will be better understood by referring to the following detailed description. Unless otherwise stated, the same reference numerals denote the same elements throughout the drawings and written description, and therefore their description will not be repeated. In the drawings, the relative sizes of elements, layers, and regions may be exaggerated for clarity. Detailed Implementation

[0036] In the following description, exemplary embodiments will be presented in more detail with reference to the accompanying drawings. However, the invention may be embodied in various different forms and should not be construed as limited to the embodiments shown herein. Rather, these embodiments are provided as examples to make this disclosure thorough and complete, and to fully convey various aspects and features of the invention to those skilled in the art. Therefore, those skilled in the art will fully understand that processes, elements, and techniques not essential to various aspects and features of the invention may not be described.

[0037] Figure 1This is a block diagram illustrating a system including a memory device configured to store information provided by an application (e.g., on a computer). An example of a memory device is a solid-state drive (SSD), where the SSD uses block storage technology to store data. In block storage devices, raw data is stored at a specific device address within the memory device, and the application provides the device block offset (often called Logical Block Addressing (LBA)) and the length of the data to be stored on the device. On the other hand, applications using object storage understand key-value semantics. Therefore, a converter is needed to map keys in object storage to value locations (e.g., LBAs) in the block storage device to perform operations such as writing, reading, and deleting data. Thus, when an application (e.g., a software application) wants to retrieve a stored object, it can provide the key to the memory device, and the converter can map the key to the value location to retrieve the object from the memory device based on the key. Similarly, when an application wants to delete a stored object, it can provide the key corresponding to the object, and the converter can map the key to the value location and delete the object. Therefore, by using a converter, keys used by object storage applications can be mapped to value locations on block storage devices to perform device operations. However, performing such a conversion is typically resource-intensive, thus slowing down the overall system, and performance can be negatively impacted as memory device size scales. Therefore, a more efficient technique that does not degrade performance during size scaling is desired.

[0038] Various embodiments of this disclosure describe key-value to block storage conversion techniques during put (e.g., storage), get (e.g., retrieval), and delete operations between applications and storage devices. In some embodiments, key-value to block storage conversion can be used during crash recovery or system restart operations. Figure 1This is an example block diagram of system 100 (such as a computer including at least a processor 106 coupled to a memory device 114, such as an SSD for storing data). In some embodiments, processor 106 includes application 102, which communicates with memory 112 using block storage via I / O interface 104 of processor 106 and I / O interface 108 of memory device 114. Controller 110 may be coupled to memory 112 and may be configured to regulate data traffic flowing between memory 112 and application 102. An example of memory device 114 is an SSD, and an SSD may be divided into multiple segments of equal size. For example, a 16TB SSD may be divided into approximately 4 billion segments of equal size (4KB). However, an SSD may be divided into segments of larger or smaller size based on application or end-user preferences.

[0039] According to an embodiment, an object to be stored by application 102 has a key, and a hash function is applied to the key to generate a hash. The hash can then be used to generate an index, and the object can be stored as a value in a segment of a memory device corresponding to the generated index. The location of the segment within the memory device can be determined by the index generated from the hash. For example, a hash function can be applied to the key to generate a hash (e.g., a 256-bit number), and the index can be generated based on that hash. The index can then be used (among 4 billion segments) to find a segment indicator that matches the index. For example, the segment indicator can be the address or location of the segment within the device, and if a segment indicator that matches the index is found, the object can be stored in that segment. Similarly, if it is desired to retrieve an object stored in a segment, the key corresponding to the object to be retrieved can be hashed to generate an index, and the object can be retrieved from the segment based on the index. Therefore, hash-based key-value conversion techniques according to various embodiments of this disclosure provide a method to reduce resource consumption and further improve performance during size scaling. However, in practice, this is not as simple as it sounds. For example, an index generated from a hash may not correspond to a segment because the index may exceed the range of available segments. That is, if there are 4 billion segments, any index number greater than 4 billion will fall outside that range. Furthermore, in some cases, the segment corresponding to the generated index may already be occupied by an object corresponding to another key, leading to object conflicts. Therefore, embodiments of this disclosure can implement the various index generation and conflict resolution techniques described herein.

[0040] Figure 2This is a flowchart of a method for finding available slots on a memory device according to an embodiment. In some embodiments, when an application expects to write an object to a memory device (e.g., an SSD), various steps are performed to place the object onto the SSD. For the purposes of this disclosure, and to facilitate the description of various embodiments of this disclosure, a 16TB SSD divided into equal-sized 4KB segments will be used as an example memory device to illustrate the embodiments. Therefore, the 16TB SSD is divided into approximately 4 billion segments, and each segment has an associated indicator (e.g., an address or locator). In some embodiments, each segment includes metadata corresponding to that segment, wherein the metadata may indicate, for example, whether the segment contains data or the segment is empty. Examples of metadata are provided in... Figure 7 As shown in the figure. In some embodiments, a segment of the memory device can be retained as a segment vector, and within the segment vector, there can exist a meta-vector that can be used to store metadata. The meta-vector is similar to the metadata stored in each segment, but is designed to provide metadata information quickly without having to read the metadata information in each segment.

[0041] According to one embodiment, the key corresponding to the object is hashed to generate a value (202). Various hashing techniques are well known in the art, and some hashing techniques may be more preferred than others. The examples described in this disclosure use the Secure Hash Algorithm 256 (SHA-256); however, those skilled in the art will find that other suitable hashing techniques can be used instead. In some embodiments, other techniques known to those skilled in the art can be used to generate an index from the hashed key. Here, the SHA-256 hash generates a 256-bit number of bits as the actual hash value used to generate an index (e.g., a first index). In some embodiments, the hash-generated index is used to identify a segment on the SSD with a corresponding indicator (e.g., an address), and the object is stored in that segment. However, in some cases, for example, because there are more potential index values ​​than segments, the index generated by the hash function may not correspond to a segment. Therefore, some indexes may fall outside the range of segments available on the SSD. The number of indexes that can be generated may vary depending on the type of hashing technique used. To illustrate, the index can be a 256-bit number generated using hashing techniques, while the segments can have 32-bit indicators. Therefore, the value of the segment indicator can range from 0 to 2. 32 The range is -1, while the first index generated from the hash value can be between 0 and 2. 256 Between -1 and -1. In other cases, for example, because a segment may already be occupied by another object based on a different key, the index may correspond to an unavailable segment. Therefore, embodiments of this disclosure provide a technique for more efficiently finding the corresponding segment that can be used to store an object.

[0042] Therefore, after hashing the first key, a 256-bit index (e.g., the first index) is generated based on the hash of the first key (202), and a determination is made regarding whether the first index is within the range of the corresponding segment indicator (e.g., [0…2). 32 The determination within [-1]) (204). If the first index is not within the range of the corresponding segment indicator, another index (e.g., a second index) is generated based on the same hash value of the first key to determine whether the second index falls within the range of the segment indicator. As mentioned above, the first index is the actual hash value generated by the hash function, while the second index is a new value generated based on the hash value. Therefore, according to one embodiment, the second index can be generated by extracting the first bit from each byte of the 256-bit hash to generate a new index (e.g., the second index) (214). That is, the 256-bit hash has 32 bytes, and the first bit of each of the 32 bytes is extracted and combined to generate a new 32-bit index (e.g., the second index). If the second index falls within the range of the segment indicator, the metavector of the segment corresponding to the second index is read (206) to determine whether the segment is empty or occupied by another object (208). If the segment is empty, the segment is available for the object to be written. On the other hand, if a segment is occupied by another object, this indicates a conflict where multiple objects (each corresponding to a different key) share a common index. In this case, information about the conflict can be stored in metadata to facilitate conflict resolution.

[0043] According to an embodiment, the metadata of the segment in which the first object is stored includes a conflict table, which may contain information related to a second object that has conflicted with the first object (e.g., a common index corresponding to the segment has been generated for both the first and second objects). For example, the conflict table (or other data structure) may be an array of information including the offset of the second object that conflicts with the first object at the segment (e.g., the distance between this segment and another segment storing information related to the second object) and a hash of the key corresponding to the second object. Therefore, if no conflict occurs, the conflict table can be empty, but whenever a conflict occurs, the conflict table can be updated to include an array of information related to the second object that conflicts with the first object in that segment. For example, the first object corresponding to the first key is stored in the segment. At this time, the conflict table is still empty because the segment was once available, so there is no conflict. Then, the index corresponding to the second key used for the second object points to the same segment in which the first object is stored. This causes a conflict because the segment has already been occupied by the first object. In this case, the second object will be stored in a different segment, and the conflict table can be updated with information about the actual location of the second object. Therefore, when it is later expected that a second object will be retrieved and the second index used for the second key points to the segment containing the first object, the conflict table can point to the actual location of the second object. Further details regarding the retrieved object will be described later.

[0044] Returning to step (208), a conflict exists when a determination is made that the selected segment already contains an object. In this case, if this is the first conflict, the conflict table does not yet exist, and therefore the conflict table is empty. If this is not the first conflict (e.g., a conflict has occurred before), the conflict table likely already exists. If the conflict table already exists, a determination is made as to whether the conflict table has been read (210). If the conflict table has not yet been read or is still empty, metadata is further read to determine which key is associated with the object stored in the segment, and if this information is not in the conflict table, the conflict table can be updated with this information (226). Next, the hash read from the metadata or the conflict table (if the conflict table already exists) is compared with the hash of the current key, and if they match, this is an overwrite condition (232). The handling of the overwrite condition will be described later.

[0045] While performing steps (226) and (232), a determination is made (212) as to whether the index (e.g., the first index) determined in step (204) is within the range of the segmentation indicator. If the index is within the range, the first bit is extracted from each byte of the 256-bit hash to generate a new index (e.g., the second index) (214). If the index is not within the range, since the second index has already been generated in step (214), a third index is generated by extracting the second bit from each byte of the 256-bit hash (i.e., based on the hash value) to generate another index (216).

[0046] In some embodiments, if the index generation from extracting bits from each byte has been performed eight times, all 8 bits of each byte of the 256-bit hash are exhausted, and no more bits are available to generate a new index. Therefore, if all bits are exhausted (218), segments on the memory device are probed one segment at a time (220, 222). More specifically, the probing process can be performed by generating a first index again from each byte to establish a starting point (the same as the first index generated in step (202)). The metavector of each segment is then probed until a free segment is found (222). Once a free segment is found (224), the metadata corresponding to the free segment is read (226), and a determination is made regarding whether the segment should be overwritten as described above (232, 234) by comparing the index of the segment found by probing with the metadata or conflict table. If they match, this is an overwrite condition, but if they do not match, this is not an overwrite condition. If this is not an overwrite condition and there are no available empty slots (because the probes have been exhausted), the write fails because there is no space (228). On the other hand, if this is an overwrite condition, a determination is made as to whether the segment is large enough to write the object to that segment (250). If the segment is not large enough, there is not enough space in the segment and the write fails (228). If the segment is large enough, the overwrite can proceed because the object can be contained in the segment, and the transaction can be processed, and the metadata and object can be written to the segment (246).

[0047] Returning to step (216), once a new index is generated by extracting the second bit from each byte, the metavector in the corresponding segment is read (206) to determine whether the segment is empty or occupied (208). In some embodiments, this process is repeated until an empty segment is found.

[0048] In some embodiments, once available segments are identified, the total number of segments required to store the object is determined based on its size. For example, if the object size is 8KB and each segment is 4KB, two segments are needed to store the object. In some embodiments, a free list is consulted to determine which segment and its corresponding free slot are located. For example, if two segments are needed to write an 8KB object and these two segments are adjacent, the entire object can be written in a single disk write.

[0049] In some embodiments, the free list may be available on a portion of the memory device. The free list may be a list showing the location of segment offsets and the number of free segments, and the free list is continuously updated with the status of available segments. Thus, once the total number of required free segments is calculated in step (230), the free list can be queried to determine the exact matching segments based on the size of the object to be written (e.g., 4KB segments for a 4KB object, or two 4KB segments for an 8KB object) (236) and to obtain these free segments such that the modified number of segments is within the maximum threshold of disk writes (e.g., a single disk write size limit) (238). In other words, the free list optimizes the best way to use available segments with fewer disk writes (e.g., a single disk write) by determining the best possible solution, which is the case where multiple segments are sequentially. Next, a determination is made as to whether the required free segments have been found (240). If segments are found, a determination is made as to whether the conflict or the modified segment is not within the maximum single disk write size limit (242). In other words, if multiple segments are sequential or within the granularity of a single disk write, data writing can be performed in a single write and no transaction is required. However, if multiple segments are not sequential or not within the granularity of a single disk write, a transaction (i.e., more than one disk write) is required to write multiple segments, and therefore it is not within the maximum single disk write limit. If a conflicting or modified segment is not within the segment size, a transaction for metadata is required (248). If a conflicting or modified segment is within the segment size, no transaction is required (244).

[0050] Returning to step (240), if it is determined that no segment is found, a determination is made regarding whether this is an overwrite (234). If this is not an overwrite, the write process fails (228). If this is an overwrite, a determination is made regarding whether the segment is large enough based on the size of the object (250). If the segment is not large enough, the segmentation fails (228). If the segment is large enough, the transaction is processed, and metadata and objects can be written to the segment because no free segment is available (246). Therefore, Figure 2The flowchart can be used to determine whether a transaction is needed for metadata writing (248), whether a transaction is needed for both metadata writing and object writing (246), or whether a transaction is not needed (244).

[0051] Figure 3 This is a flowchart of a method for placing an object into one or more slots of a storage device once an available slot is identified. According to an embodiment, based on... Figure 2 The result selects one of two paths: whether a transaction is needed (302). For example, if it is determined that a transaction is not needed, then, for example, based on... Figure 2 In step (238), the free slots found are sequential, and a determination is made regarding whether metadata and objects can be written to multiple sequential segments (304). If metadata and objects can be written to sequential segments, both metadata and objects are written simultaneously (312), the segment vector is updated (310), and the writing process ends (314). In some embodiments, the segment vector may be a portion of the storage device (e.g., at the beginning of the SSD) and may include a bit-based indicator of the status of each segment. For example, the indicator may be a two-bit indicator representing the status of each segment as available, metadata, data chunk (or object), or conflicting entry. Thus, the status of any segment on the storage device can be quickly determined by referencing the metadata vector. Because each segment is represented by only two bits, an example 16TB SSD with 4 billion segments consumes approximately 950MB of storage space. In some embodiments, the free list is also referenced and updated accordingly.

[0052] Returning to step (304), if it is determined that the metadata and object cannot be written to multiple sequential segments, the object is written separately (306), followed by the metadata (308). In some embodiments, if the object cannot be written to multiple sequential segments, it can be written in several parts, for example, the first half of the object can be written, followed by the second half, and then the metadata. Next, the segment vector is updated (310), and the writing process ends (314).

[0053] Returning to step (302), if it is determined that a transaction is needed, then make a transaction regarding whether it is needed only for metadata writing (such as in...). Figure 2 The step (248) determined still requires transactions for both metadata writing and object writing (such as in Figure 2The determination (316) is made in step (246). If it is determined that the object part is not needed (i.e., only metadata writing is needed), a determination is made regarding the segmentation of the order in which metadata and objects can be written (318). If it is determined that the segmentation of the order in which metadata and objects can be written cannot be, the object is written (320), and then the metadata is written (322). If it is determined that the segmentation of the order in which metadata and objects can be written can be, both metadata and objects are written simultaneously (324). Next, based on whether there has ever been a conflict (e.g., as in...), Figure 2 In step (208), when determining whether a segment is not empty, one of two paths is selected (326). If a conflict exists, the old metadata from the first conflicting entry (e.g., when in) Figure 2 In step (208), the metadata corresponding to the object already stored in the segment when the conflict is determined is added to the meta-transaction object (328), and then the old segment vector region to be modified is added to the meta-transaction object (330). On the other hand, if there is no conflict, step (328) is skipped, and the old segment vector region to be modified is added to the meta-transaction object (330). Next, the meta-transaction object can be written to a predefined area of ​​the memory device (332). Next, based on whether a conflict has ever existed (e.g., as in...), Figure 2 In step (208), when determining whether a segment is not empty, one of two paths is selected (334). If there is no conflict (334), the segment vector is updated (338), and then the meta-transaction object is deleted (340). On the other hand, if there is a conflict, the conflict table is updated (336) with the offset or other position indicator of the segment storing the metadata of the object just written, the segment vector entry is written by updating with the entry corresponding to the segment just modified (338), and then the meta-transaction object is deleted once the write is successful (340). Accordingly, the write operation is complete (314).

[0054] Returning to step (316), if a determination is made regarding the portion of the object to be written for the transaction, the old object corresponding to the key is read (342). Then, the old object, along with the metadata on the old disk (ondisk), is added as part of the data transaction object (344). The data transaction object can then be written to a predefined area of ​​the storage device (346). Next, the metadata and object of the key are updated (348), and the data transaction object is deleted (350). Accordingly, the write operation is complete (314).

[0055] Figure 4This is a flowchart of a method for retrieving an object from a memory device. According to an embodiment, when an application expects to retrieve an object from a memory device, the application can provide a key corresponding to the object, and the object can be retrieved from the memory device. Similar to a put operation, in a get operation, a key-to-object storage conversion is performed between the application requesting the object and the memory device storing the object. In some embodiments, the key corresponding to the object is hashed to generate a first index based on the hash of the key (402). In some embodiments, the same hashing technique used during the get operation is the same as the hashing technique used when writing the object. Accordingly, the SHA-256 hashing technique can be used. However, other hashing techniques may also be applicable, as long as the same hashing technique is used for writing the object. In some embodiments, other techniques known to those skilled in the art can be used to generate the index from the hashed key. Once the first index is generated, a determination is made regarding whether the first index is within the range of a corresponding segmentation indicator of the memory device (404). If the first index is not within the range of the corresponding segment indicator, a first bit is extracted from each byte of the 32 bytes of the hash, and then the first bit is combined to generate a new 32-bit index (e.g., a second index) (412). In some embodiments, if the generated first index is within the range of the corresponding segment indicator, a check for the existence of a metavector for the segment corresponding to the first index is applied (406). In some embodiments, this determination can be made by checking a bit-based status indicator indicating the existence of metadata (408). If the metadata does not exist, the retrieval operation fails because no object corresponding to the hashed key was found (414). If the metadata exists, the metadata is read (410), and a determination is made regarding whether the hash of the first key matches or does not match the stored hash in the metadata (416). If the hash matches and the object is in the same segment as the metadata (420), the object can be retrieved, and the retrieval is successful (424). If the object is not in the same segment as the metadata, a determination is made regarding whether the data blocks of the object are stored sequentially (426). If the objects are stored sequentially, the entire object is read and retrieved in a single read (428). If the objects are not stored sequentially, the objects are read in multiple reads because they are scattered across multiple offsets (e.g., non-contiguous segments) (430).

[0056] Returning to step (416), if the hash of the first key does not match the stored hash in the metadata, the conflict table entry is queried to check if any matching hash key exists (418). If the conflict table does not provide a segment, the retrieval process fails because no object is found (414). If the conflict table contains an indicator indicating which segment the object is in, the metadata at that corresponding segment is read, and then, as previously described, a determination is made regarding whether the object is in the same segment as the metadata (420). Thus, the retrieval operation can be performed efficiently even in the presence of a conflict because the conflict table in the metadata is updated during the put operation. Therefore, when a conflict exists during retrieval, the conflict table can be used to determine which segment the object is actually in.

[0057] Figure 5This is a flowchart of a method for deleting an object from a memory device. According to an embodiment, the process for deleting an object is similar to a retrieval operation. Therefore, the key corresponding to the object to be deleted is hashed to generate a first index based on the key hash (502). In some embodiments, the same hashing technique used during the deletion process is the same as the hashing technique used when writing the object. Therefore, the SHA-256 hashing technique can be used. However, other hashing techniques may also be applicable, as long as the same hashing technique is used for writing the object. Once the first index is generated, a determination is made regarding whether the first index is within the range of a corresponding segmentation indicator on the memory device (504). If the first index is not within the range of the corresponding segmentation indicator, a first bit is extracted from each byte of the 32 bytes of the index, and then the first bit is combined to generate a new 32-bit index (e.g., a second index) (512). In some embodiments, if the generated first index is within the range of the corresponding segmentation indicator, the system checks for the existence of a meta-vector for the segment corresponding to the index (506). In some embodiments, this determination can be made by checking a bit-based status indicator indicating the existence of metadata (508). If the metadata does not exist, the deletion process fails because no object corresponding to the hashed key was found (514). If the metadata exists, the metadata is read (510), and a determination is made regarding whether the hash of the first key matches or does not match the stored hash in the metadata (516). If the hash matches and the conflict table is empty, there is no conflict, and the object in the corresponding segment can continue to be deleted (520). Next, a determination is made regarding whether the segment vector can be written in a single write (528). If the segment vector cannot be written in a single write, a transaction is started (530). If the segment vector can be written in a single write, the transaction start is skipped, and the disk meta-value size or conflict table is updated as needed (532). Next, the metadata and object segments in the segment vector are updated to indicate that they are now available (534). Next, a determination is made regarding whether the write is part of a transaction (536). If the write is not part of a transaction, the deletion process ends (540). If the write is part of a transaction, the transaction ends first (538), and then the deletion process ends (540).

[0058] Returning to step (524), if the conflict table is not empty, the value size indicator in the metadata can be set to zero (524). Here, a value size of zero indicates that the object stored in the segment has been deleted, and therefore should not be read during future delete or retrieve operations. Once the value size is set to zero, the corresponding object(s) in the corresponding segment(s) indicated by the metadata can be deleted.

[0059] In some embodiments, when the metadata is read in step (510) and the hash of the first key does not match the stored hash in the metadata and is therefore inconsistent (516), the conflict table is read to determine which segment contains the actual metadata corresponding to the object (518), and that metadata is read (522). If the hash stored in the metadata matches the hash of the key, the number of entries in the conflict table is decremented by one (526), ​​and the object can be deleted. If the hash stored in the metadata does not correspond to the hash of the key, the deletion process fails (514).

[0060] Figure 6 This is a flowchart of a method for recovering from a system crash or restart. When a system or application crashes during disk operations (such as write operations), data may be corrupted. Therefore, a method for recovering data is described according to embodiments. In some embodiments, when the system restarts after a crash, a predefined disk space for transaction objects is read to determine if any pending or incomplete transactions exist (604). If any pending transactions exist, the transactions are executed, and metadata and objects are recovered (620). Once completed, the transaction objects can be deleted (622). If no pending transactions exist, steps (620) and (622) are skipped, and a segment vector (606) is read, and a free list (608) and a meta vector (610) are formed in memory. If caching is enabled to store conflict tables (612), the corresponding metadata is read from the cache (614), and the conflict table corresponding to that metadata is also read from the cache (616), and this information is added to the conflict table and metadata stored in the cache (618). Therefore, even if the system or application crashes, any pending transactions can be recovered and restored upon restart. Thus, the hash-based key-value conversion techniques according to various embodiments of this disclosure provide a method for reducing resource consumption and further improving performance during size scaling.

[0061] It should be understood that although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the spirit and scope of the invention, the first element, first component, first region, first layer, or first portion described below may be referred to as a second element, second component, second region, second layer, or second portion.

[0062] It should be understood that when an element or layer is referred to as being "located" on, "connected to," or "coupled to" another element or layer, that element or layer may be directly located on, connected to, or coupled to the other element or layer, or there may be one or more intermediate elements or layers. Furthermore, it should be understood that when an element or layer is referred to as being "between" two elements or layers, that element or layer may be the only element or layer between the two elements or layers, or there may be one or more intermediate elements or layers.

[0063] The terminology used herein is for describing particular embodiments and is not intended to limit the invention. As used herein, the singular forms “a” and “an” are intended to equally include the plural forms unless the context clearly indicates otherwise. It should also be understood that, when used in this specification, the terms “comprise,” “comprising,” “include,” and “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Descriptions such as “at least one of…” preceding a list of elements modify the entire list of elements, not individual elements within the list.

[0064] As used herein, the terms “substantially,” “approximately,” and similar terms are used as approximate terms rather than as terms of degree, and are intended to describe the inherent biases in measured or calculated values ​​that will be recognized by one of ordinary skill in the art. Furthermore, when describing embodiments of the invention, the use of “may” refers to “one or more embodiments of the invention.” As used herein, the terms “use,” “using,” and “used” can be considered synonyms for the terms “utilize,” “utilizing,” and “utilized,” respectively.

[0065] Electronic or electrical devices and / or any other related devices or components according to embodiments of the invention described herein can be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuits), software, or a combination of software, firmware, and / or hardware. For example, various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. Furthermore, various components of these devices can be implemented on flexible printed circuit films, tape carrier packages (TCPs), printed circuit boards (PCBs), or formed on a substrate. Additionally, various components of these devices can be processes or threads running on one or more processors in one or more computing devices, executing computer program instructions, and interacting with other system components to perform the various functions described herein. The computer program instructions are stored in memory, which can be implemented in the computing device using standard memory devices such as, for example, random access memory (RAM). The computer program instructions can also be stored on other non-transitory computer-readable media, such as, for example, CD-ROMs, flash drives, etc. Furthermore, those skilled in the art will recognize that, without departing from the spirit and scope of the exemplary embodiments of the invention, the functionality of various computing devices can be combined or integrated into a single computing device, or the functionality of a particular computing device can be distributed across one or more other computing devices.

[0066] It should be understood that any component or any combination of components described throughout this disclosure can be used to perform one or more operations described in the flowchart. Furthermore, the described operations are merely example operations and may involve various additional steps not explicitly described herein. Additionally, the temporal order of the operations may vary.

[0067] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms (such as those defined in common dictionaries) should be interpreted as having the same meaning as they have in the relevant field and / or the context of this specification, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0068] The embodiments described herein are merely examples. Various alternative embodiments will be apparent to those skilled in the art from the specific disclosed embodiments. These alternative embodiments are also intended to fall within the scope of this disclosure. Therefore, the embodiments are defined only by the appended claims and their equivalents.

Claims

1. A method comprising: Calculate the first index based on the hash of the first key used for the first object to be stored in the memory device; Determine the availability of the first segment corresponding to the first index on the memory device; In response to determining that the first segment corresponding to the first index is not available for storage, a second index is computed based on the hash of the first key used for the first object; Determine the availability of the second segment on the memory device corresponding to the second index; as well as Add the position indicator of the second segment to the conflict table in the first metadata of the first segment. The first metadata of the first segment in which the second object is stored includes the conflict table, which contains information related to the first object that has conflicted with the second object, and the conflict table is an array of hash information including the offset of the first object that conflicts with the second object at the first segment and the first key corresponding to the first object.

2. The method according to claim 1, further comprising: In response to determining that the second segment is available, the first object is written to the second segment.

3. The method according to claim 2, wherein, The calculation of the second index includes: extracting bits from each byte of the hash to form the second index.

4. The method according to claim 2, wherein, Determining the availability of the first segment includes: reading a status identifier from the first metadata, wherein the status identifier indicates a status corresponding to the first segment being unavailable for storage.

5. The method according to claim 4, further comprising: In response to the collision table being empty, the association between the hash of the first key and the indicator of the position of the second segment is written into the collision table.

6. The method according to claim 1, wherein, The calculation of the first index includes using a secure hash algorithm 256.

7. The method according to claim 4, further comprising: Receive a request indicating the acquisition of the first key; The first index is calculated based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is inconsistent with the stored hash stored in the first metadata; Read the conflict table in the first metadata to determine that the first object is stored in the second segment; as well as In response to reading the conflict table, the first object is retrieved from the second segment.

8. The method according to claim 4, further comprising: Receive a deletion request that identifies the first key; The first index is calculated based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is inconsistent with the stored hash stored in the first metadata; Read the conflict table in the first metadata to determine that the first object is stored in the second segment; as well as In response to reading the conflict table, the first object is deleted from the second segment.

9. The method according to claim 4, further comprising: Receive a deletion request that identifies the first key; The first index is calculated based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is consistent with the stored hash stored in the first metadata; The conflict table in the first metadata is determined to include an indicator of the location of the second segment; Set the size of the conflict table to zero; as well as Delete the first object from the second segment.

10. The method of claim 4, further comprising: Receive a deletion request that identifies the first key; Calculate the first index based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is consistent with the stored hash stored in the first metadata; In response to determining that the hash of the first key matches a stored hash stored in the first metadata, the collision table in the first metadata is determined to be empty; as well as In response to reading the conflict table, the first object is deleted from the second segment.

11. An object storage system configured to store objects, the object storage system comprising a segmented memory device, the memory device being configured to: Calculate the first index based on the hash of the first key used for the first object to be stored in the memory device; Determine the availability of the first segment corresponding to the first index on the memory device; In response to determining that the first segment corresponding to the first index is not available for storage, a second index is computed based on the hash of the first key used for the first object; Determine the availability of the second segment on the memory device corresponding to the second index; as well as Add the position indicator of the second segment to the conflict table in the first metadata of the first segment. The first metadata of the first segment in which the second object is stored includes the conflict table, which contains information related to the first object that has conflicted with the second object, and the conflict table is an array of hash information including the offset of the first object that conflicts with the second object at the first segment and the first key corresponding to the first object.

12. The system according to claim 11, wherein, The memory device is also configured to write the first object into the second segment in response to determining that the second segment is available.

13. The system according to claim 12, wherein, Calculating the second index involves extracting bits from each byte of the hash to form the second index.

14. The system according to claim 12, wherein, Determining the availability of the first segment includes: reading a status identifier from the first metadata, wherein the status identifier indicates a status corresponding to the first segment being unavailable for storage.

15. The system according to claim 14, wherein, The memory device is further configured to: in response to the collision table being empty, write the association between the hash of the first key and the indicator of the position of the second segment into the collision table.

16. The system according to claim 11, wherein, Calculating the first index involves using a secure hash algorithm 256.

17. The system according to claim 14, wherein, The memory device is also configured to: Receive a request indicating the acquisition of the first key; The first index is calculated based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is inconsistent with the stored hash stored in the first metadata; Read the conflict table in the first metadata to determine that the first object is stored in the second segment; as well as In response to reading the conflict table, the first object is retrieved from the second segment.

18. The system according to claim 14, wherein, The memory device is also configured to: Receive a deletion request that identifies the first key; Calculate the first index based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is inconsistent with the stored hash stored in the first metadata; Read the conflict table in the first metadata to determine that the first object is stored in the second segment; as well as In response to reading the conflict table, the first object is deleted from the second segment.

19. The system according to claim 14, wherein, The memory device is also configured to: Receive a deletion request that identifies the first key; Calculate the first index based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is consistent with the stored hash stored in the first metadata; The conflict table in the first metadata is determined to include an indicator of the location of the second segment; Set the size of the conflict table to zero; as well as Delete the first object from the second segment.

20. The system according to claim 14, wherein, The memory device is also configured to: Receive a deletion request that identifies the first key; Calculate the first index based on the hash of the first key; Read the first metadata of the first segment corresponding to the first index; In response to reading the first metadata, it is determined that the hash of the first key is consistent with the stored hash stored in the first metadata; In response to determining that the hash of the first key matches a stored hash stored in the first metadata, the collision table in the first metadata is determined to be empty; as well as In response to reading the conflict table, the first object is deleted from the second segment.

21. A method comprising: Receive the request to retrieve the first key; Calculate the first index based on the hash of the first key used to retrieve the object from the storage device; Read the first metadata of the first segment corresponding to the first index; In response to reading metadata, it is determined that the hash of the first key is inconsistent with the stored hash stored in the first metadata; Read the conflict table in the first metadata to determine if the object is stored in the second segment; as well as In response to reading the conflict table, the object is retrieved from the second segment. The first metadata of the first segment in which another object is stored includes the conflict table, which contains information related to the object that has conflicted with the other object, and the conflict table is an array of hashes including the offset of the object that conflicts with the other object at the first segment and the first key corresponding to the object.

22. The method according to claim 21, wherein, The collision table includes a second index based on the hash of the first key and an offset of the object stored in the first segment, wherein the offset is an association between the hash of the first key and an indicator of the position of the second segment.

Images