Weak reference to an assigned logical cluster

Abstract

Aspects of the present disclosure relate to weak references of allocated logical clusters. In an example, a deduplication hash table stores associations between block hashes and block addresses, such that blocks can be identified and used in place of another block storing the same data. Weak references can be generated on blocks, such that a perfect weak reference indicates that the block has not been changed. When a block is identified as having a hash that matches a hash of an existing block in the hash table, it can be determined whether the weak reference on the existing block is perfect. If there is no perfect weak reference on the existing block, then the existing block can have been changed such that it cannot be used for deduplication. In contrast, if the weak reference is perfect, then the existing block has not been changed such that it can be correspondingly used for deduplication.

Description

Background Technology

[0001] Identifying and processing redundant data stored by the file system is resource-intensive and can introduce additional complexity to read and write operations. For example, various deduplication techniques can increase code complexity, reduce file system performance, and / or, if improperly implemented, even cause data corruption, as well as other losses.

[0002] Embodiments have been described with reference to these and other general considerations. Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments are not limited to solving the specific problems identified in the background section. Summary of the Invention

[0003] This disclosure relates to weak references to allocated logical clusters. In the example, a deduplication hash table is used to store the association between block hashes and block addresses, allowing the block to be later identified and used in place of another block storing the same data. Weak references can be generated on a block when an entry for that block is added to the deduplication hash table. If the block is modified, the weak references can be destroyed or otherwise removed. As a result, intact weak references can indicate that the block has not been altered and that the associated entry in the deduplication hash table is not stale.

[0004] When a block is identified as having a hash that matches the hash of an existing block in the hash table, it can be determined whether the weak reference on the existing block is intact. If the weak reference is no longer intact (e.g., the weak reference has been broken or removed), the data stored by the existing block may have been altered, making the existing block unsuitable for deduplication. Conversely, if the weak reference is intact, the data stored by the existing block has not been altered, allowing it to be used for deduplication.

[0005] This summary portion is provided to introduce the selection of a simplified form of concept further described in the detailed embodiments below. This summary portion is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Attached Figure Description

[0006] Non-restrictive and non-exhaustive examples are described with reference to the following figures.

[0007] Figure 1 An overview of an example system in which weak references to logical clusters, according to the aspects described herein, can be used to perform data deduplication is shown.

[0008] Figure 2A An overview of example methods for processing file system blocks based on weak references to allocated logical clusters is shown.

[0009] Figure 2BAn overview of example methods for performing data deduplication on file system blocks based on weak references to allocated logical clusters is shown.

[0010] Figure 3 An overview of example methods for writing data to a file system block according to aspects of this disclosure is shown.

[0011] Figure 4 An overview of example methods for restoring the state of a file system after a failure, according to aspects of this disclosure, is shown.

[0012] Figure 5 This is a block diagram illustrating an example physical component of a computing device in which aspects of this disclosure can be utilized.

[0013] Figure 6A and 6B This disclosure is a simplified block diagram of a mobile computing device that can be implemented using its aspects.

[0014] Figure 7 This is a simplified block diagram of a distributed computing system in which aspects of this disclosure can be practiced.

[0015] Figure 8 A tablet computing device for performing one or more aspects of this disclosure is shown. Detailed Implementation

[0016] In the following detailed description, reference is made to the accompanying drawings, which form part of the description, and specific embodiments or examples are illustrated therein by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be practiced as methods, systems, or devices. Therefore, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description should not be construed as limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0017] In this example, data deduplication techniques are used to identify and reduce the amount of redundant data stored by the file system. For example, a file can be a stream of bytes, stored as a collection of blocks or a logical cluster within the file system. In some cases, a block in a first file stores the same data as another block in a second file. Blocks with the same data do not need to be at the same file-related offset, allowing the same data to exist at a first offset in the first file and a second offset in the second file. In such cases, the file system can store the data as a single shared block instead of two blocks with the same data. The shared block can be referenced by each representation of the corresponding file. Thus, the same logical cluster is used at least partially to store data common to both the first and second files. However, it can be difficult to accurately, reliably, and efficiently identify such common blocks, update file metadata to reference the shared blocks, and further maintain multiple references to the shared blocks.

[0018] Hash tables can be used for data deduplication, where they store the hash value of a block's data and the block's location (e.g., the block's logical address or logical cluster number on the data repository, etc.). Therefore, hash tables are used to identify data already stored by the file system, allowing existing blocks to be referenced instead of retaining blocks that store duplicate data. However, maintaining an up-to-date hash table that can be used to identify common blocks can be resource-intensive, as write operations may require the hash table to be updated simultaneously. Additionally, performing "in-line" deduplication can require significant processor utilization, memory utilization, and disk bandwidth to use and maintain the deduplication hash table while operations are being handled by the file system. As another example, offline deduplication can be time-intensive because the file system blocks must first be hashed so that common blocks can be identified and subsequently replaced to reference the shared blocks accordingly.

[0019] Therefore, this disclosure relates to weak references to allocated logical clusters. As described herein, a file comprises a set of blocks such that the file has “strong” references to its constituent blocks. A strong reference indicates that the data stored by the block is at least a portion of a given file. Multiple files may each have strong references to the same blocks in a file system, while expecting changes to each copy of the file to remain distinct; such blocks are referred to herein as shared blocks. In some cases, a reference count is maintained for a given block, storing the number of strong references to that block. Thus, a shared block will have at least two reference counts, indicating that the shared block is associated with multiple files. To avoid data corruption, a shared block cannot be modified by write operations associated with any of the referenced files in the referenced files, because a write operation to one associated file would cause data corruption in a second associated file by virtue of the shared block. Instead, such write operations can instead cause different blocks to be used, thereby correspondingly maintaining the data in the shared block.

[0020] In contrast to strong references, which indicate that data stored in an associated block is being used by the file and, in some cases, prevent modification of that block, weak references allow write operations associated with the referenced block to proceed. Therefore, such a block can have a strong reference count of "1" (i.e., a single strong reference to the associated file) and can also have associated weak references. If the block is modified, the weak reference can be destroyed, while the strong reference associated with the file can be preserved. In some cases, any change to the strong reference count of a block can cause the weak reference to be destroyed or removed. As another example, an operation that decrements the strong reference count of a block causes the weak reference to be destroyed or removed.

[0021] As another example, an operation that reduces the strong reference count to below "2" destroys or removes a weak reference. For example, the strong reference count could decrease to "1," indicating that the block is no longer a shared block. As another example, the strong reference count could decrease to "0," indicating that the block can be freed. In some cases, destroying or removing a weak reference is additionally or alternatively conditional upon a modification to the block. For example, the strong reference count could decrease to below "2," and the block could be further overwritten, thereby destroying or removing the associated weak reference. While the example weak references described herein indicate the weak reference state for an associated block, it will be understood that a weak reference does not necessarily need to include a reference to its associated block.

[0022] For example, a weak reference can be an entry in a data structure that stores the association between a block (e.g., identified by a logical address or other identifier) ​​and the weak reference state for that block. The weak reference state represents the state of the block when the weak reference was created, making the weak reference intact only if the block has not been modified. Therefore, if the state of the block is about to change (e.g., as a result of an in-place write), the weak reference is first destroyed or removed to indicate that the block is no longer in the state it was in when the weak reference was created.

[0023] As described above, a deduplication hash table can be used, which stores the hash value of the data stored by a block in association with the block's location. It will be understood that any of a variety of techniques can be used to hash the block data, including but not limited to secure hash algorithms (SHA), such as SHA-1 or SHA-2, or message digest (MD) algorithms, such as MD5. The hash table can then be used to identify and subsequently use shared blocks to represent files stored by the file system instead of blocks storing duplicate data.

[0024] According to aspects of this disclosure, when a mapping is added to a hash table, a weak reference can be taken on the associated block. In some cases, the block can be locked during hashing, or, as another example, a weak reference can be taken before hashing, and subsequently verified to be intact after hashing, thus confirming that the hash and the associated weak reference represent the data stored by the block at that time. As a result, when a new block with a hash that matches an entry in the deduplication hash table is identified, the existing block associated with that hash is evaluated to determine whether the weak reference is intact or has been corrupted. In the case of a corrupted (or nonexistent) weak reference, it is no longer guaranteed that the existing block stores the same data, so that the existing block will not be used to replace the new block. In contrast, if the weak reference is intact, it is guaranteed that the data in the block has not been altered, so that the existing block can be a suitable replacement for the new block. As a result, a strong reference can be generated on the existing block and used to replace the new block, thus producing a shared block. Therefore, the existence of a valid weak reference indicates that the data at the associated block has not been altered, so that the block can be used to perform deduplication.

[0025] Compared to alternative online deduplication techniques, hash tables do not need to be maintained concurrently with file system operations, thus reducing the overhead associated with deduplication. Furthermore, unlike alternative offline deduplication techniques, the new hash table does not need to be generated for each offline processing instance, as it can instead determine which blocks have been modified and, through expansion, which entries in the deduplication table are no longer valid. Moreover, deduplication can be performed online (e.g., while the file system is in use and / or write operations are occurring), because robust weak references confirm that data stored by existing blocks has not been altered.

[0026] Compared to techniques that allow deduplication to remain hash-free, using intact weak references to blocks reduces the amount of time blocks are locked. Similarly, the file system itself does not need to be in a read-only state. As another example, a block could alternatively be rehashed immediately before deduplication to ensure that the block has not been altered. However, hashing can be computationally expensive, especially compared to determining whether a weak reference is intact according to the aspects described herein. Therefore, instead of preventing or identifying changes to blocks via immutability or rehashing, intact weak references ensure that the block has not been altered (e.g., since the block was hashed), making the block usable for deduplication and correspondingly shared among multiple files. Furthermore, such alternatives can negatively impact performance (e.g., preventing writes to certain blocks or increased disk utilization due to rehashing), or, without them, could lead to altered blocks being incorrectly used for deduplication, resulting in data corruption.

[0027] In some cases, data structures used by the file system to maintain reference counts for strong references can also be used to take and otherwise maintain weak references. For example, a file system can use two bytes to represent a reference count for a block, where at least a subset of the bits other than those used for strong reference counting can be used for weak references. For instance, 13 of the 16 bits can be used to track the number of strong references, while the higher three bits can be used for weak references. Weak references can have various states, which can be represented by a subset of bits associated with the weak reference. Example states include, but are not limited to, a state indicating no weak reference, a state indicating a valid weak reference, and a state indicating a broken weak reference. In the example where three bits are used, each bit can have an associated state (e.g., "100" indicates no reference, "010" indicates a valid or intact reference, and "001" indicates a broken reference). As another example, two bits can be used, where "10" indicates no reference, "01" indicates a valid reference, and "11" indicates a broken reference. In other cases, the data structure may alternatively or additionally represent the number of weak references (e.g., similar to the strong reference counting described herein). Therefore, weak references can be represented using any of a variety of techniques.

[0028] It will be understood that while corrupted reference states are described in this document, alternative examples may omit corrupted reference states and may instead use states indicating the absence of weak references. However, corrupted weak references can be helpful in maintaining deduplicated hash tables because they provide an indication that stale entries exist in the deduplicated hash table, allowing those entries to be identified and updated or removed. Alternatively, cleanup operations can be performed to identify and remove stale entries in the deduplicated hash table in instances with or without corrupted reference states. Deduplicated hash tables may include a bidirectional association between block hashes and block locations, allowing the hash table to be used not only to determine existing blocks based on hashes but also to identify hash table entries based on block locations. In other cases, multiple hash tables or other data structures may be used.

[0029] Figure 1 An overview of an example system 100, in which weak references to an allocated logical cluster can be used to perform data deduplication according to the aspects described herein, is shown. As shown, system 100 includes a storage system 102, a computing device 104, and a network 106. In the example, storage system 102 and computing device 104 communicate using network 106, which may include a local area network, a wireless network, or the Internet, or any combination thereof.

[0030] Storage system 102 can be a computing device, such as a server computing device or a collection of computing devices forming a distributed computing device. Computing device 104 can also be any of a variety of computing devices, including but not limited to desktop computing devices, mobile computing devices, laptop computing devices, or tablet computing devices. It will be understood that although system 100 is shown as including a storage system 102 and a computing device 104, any number of such elements can be used in other examples. Furthermore, although system 100 is shown as a client device 104 accessing storage system 102 via network 106, it will be understood that similar techniques can be utilized for storage devices local to computing device 104 in other examples. For example, at least some of the functionality described herein with respect to storage system 102 can be additionally or alternatively implemented by computing device 104.

[0031] Computing device 104 communicates with storage system 102 via network 106 to perform file operations associated with data repository 114. For example, computing device 104 can access files, modify files, or create new files stored by data repository 114. Storage system 102 includes file system manager 108, deduplication engine 110, deduplication hash table 112, and data repository 114. Therefore, file system manager 108 manages the operations performed by computing device 104 because they are related to the file system of data repository 114. For example, file system manager 108 can read blocks, allocate blocks, modify blocks in-situ, and delete blocks in data repository 114.

[0032] When blocks in data repository 114 are updated, deleted, or otherwise modified, file system manager 108 can remove associated weak references. Therefore, operations that change blocks in data repository 114 (e.g., block modification, intent to modify blocks, etc.) destroy or otherwise remove intact weak references, ensuring that intact weak references remain intact only if the block is not modified. For example, file system manager 108 can determine that a weak reference exists for a block, allowing the weak reference to be updated to a destroyed weak reference. As another example, file system manager 108 updates weak references to reflect a state indicating that no weak reference exists for that block. As described above, such aspects may include correspondingly updating the reference counting data structure.

[0033] In the example, file system manager 108 provides block cloning functionality, allowing multiple files to reference the same blocks (e.g., each using strong references). For instance, if a user of computing device 104 instructs that a target file in data repository 114 should be copied, file system manager 108 can create a second copy that includes one or more strong references to the blocks associated with the target file.

[0034] Deduplication engine 110 maintains a deduplication hash table 112. For example, deduplication engine 110 may initially process data repository 114 to generate deduplication hash table 112, where allocated blocks of data repository 114 are processed to generate a mapping of hashes to associated block locations according to the aspects described herein. Deduplication engine 110 may take weak references to blocks in data repository 114 during such processing. As an example, deduplication engine 110 utilizes the application programming interface (API) of file system manager 108 to take weak references to blocks in data repository 114, thereby causing file system manager 108 to update the reference counting data structure or other file system metadata associated with that block. As mentioned above, blocks can be locked during the hashing process (and released after the hash is generated). As another example, weak references can be taken before hashing and subsequently verified after hashing to confirm that the block was not changed during hashing. In some other examples, in addition to using the API of file system manager 108 or as an alternative, deduplication engine 110 itself may update the metadata of data repository 114.

[0035] If the deduplication engine 110 generates a hash for data stored in the first block that already exists in the deduplication hash table 112 (e.g., where an existing hash is associated with the second block), then the deduplication engine 110 can generate an instruction to the system manager 108 to replace the first block with a strong reference to the second block, thereby freeing the first block instead of retaining duplicate data in the data repository 114. In the example, the deduplication engine 110 can lock the second block and determine whether the weak references on the second block are intact before providing an instruction to the file system manager 108. The deduplication engine 110 can also lock the first block and confirm whether the weak references on the first block are intact. Therefore, as a result of locking the first and second blocks, the deduplication engine 110 is ensured that no block is modified after confirming that the associated weak references are intact. The deduplication engine 110 can unlock the block after the deduplication operation is successful (e.g., as may be indicated by the file system manager 108). It will be understood that in some other examples, in addition to or as an alternative to deduplication engine 110, file system manager 108 may lock blocks and verify associated weak references, as discussed below. If a weak reference to the first and / or second block is not intact, an indication is given not to file system manager 108 because the data at one or more blocks may have changed, making the deduplication hash table 112 no longer current.

[0036] In other cases, file system manager 108 may additionally or alternatively lock the first and second blocks before verifying their associated weak references to ensure that data stored in either block remains unchanged. Once file system manager 108 has verified the associated weak references, it can perform deduplication accordingly, replacing the first block with a strong reference to the second block and releasing the first block instead of leaving duplicate data in data repository 114. File system manager 108 can then unlock both the first and second blocks accordingly.

[0037] It will be understood that locking a block according to the aspects described herein does not require preventing changes to the file associated with the block, such that locking locks the file itself. Rather, in some examples, modification operations associated with the file can instead break the strong reference to the locked block, and this allows a new block of the file to be allocated and referenced instead (e.g., allocation-on-write operations). Thus, modification operations can succeed against the file even while the initial block is locked. As a result, the reference count of the initial block can be decremented to reflect that the file no longer has a strong reference to that block, since the file now references the newly allocated block instead. However, as a result of the deduplication operation according to the aspects disclosed herein, the initial block can eventually be referenced by another file via a strong reference, such that the reference count of the initial block is correspondingly incremented and thus remains non-zero.

[0038] As another example, or in addition to the locking techniques described above, the strong reference count associated with a block can be incremented. For example, the strong reference count for the first block and the strong reference count for the second block can each be incremented. For example, the file system manager 108 can provide an API to increment the associated strong reference count, or as another example, block cloning functionality can be used (e.g., by associating the first and second blocks with temporary files). A strong reference count of "2" or greater will cause write-on-demand allocation behavior similar to that described above to occur in response to the associated modification operation. Similarly, a strong reference count of "1" indicates that there are no other strong references to the block, making it impossible to modify the block, and the newly added strong reference also prevents the block from being deallocated. Therefore, strong references can be used as an alternative to locking to similarly ensure that blocks are not modified or deallocated, for example, during hashing and / or deduplication operations. Thus, it will be understood that although the examples in this document are described as using locking, such strong reference techniques can be similarly applied to restrict block modification during deduplication.

[0039] Therefore, the deduplication engine 110 can be a user-mode service of the storage system 102, enabling the deduplication engine 110 to utilize the API of the file system manager 108 to replace duplicate blocks using strong references to existing blocks, according to the aspects described herein. For example, the deduplication engine 110 can utilize the weak references, block cloning capabilities, and associated APIs of the file system manager 108 to maintain a deduplication hash table 112 and identify duplicate blocks in the data repository 114, allowing the duplicate blocks to be processed accordingly.

[0040] The deduplication engine 110 can also periodically or in response to one or more events generate, regenerate, or update the deduplication hash table 112. For example, the deduplication engine 110 may update the deduplication hash table 112 when resource utilization is below a predetermined threshold or according to a schedule. As mentioned above, when an associated block is modified, the file system manager 108 may update weak references to a corrupted state or may remove weak references. Therefore, the deduplication engine 110 can process new and changed blocks to update the deduplication hash table 112, such that stale entries are removed or updated (e.g., as may be identified based on corrupted weak references), and new entries are added. For example, the deduplication hash table 112 may identify changed blocks at least in part based on a snapshot taken when the deduplication hash table 112 was last updated. It will be understood that any of a variety of techniques may be used to identify changed blocks in the data repository 114 without departing from the aspects described herein.

[0041] In some cases, although the intent to modify a block is sufficient to break or otherwise remove a weak reference, the file system manager 108 may commit such changes to the data repository 114 in a manner that the block modification is not written simultaneously with the removal of a weak reference reflecting that the block has been changed. For example, the file system manager 108 may break a weak reference in memory when a write to the associated block is received or otherwise identified. The file system manager 108 may create a log entry indicating the weak reference breakage. The block may then be further modified in memory and eventually flushed to the data repository 114.

[0042] However, in the event of a failure (e.g., storage system 102 crashes, power failure occurs, or access to data repository 114 is unexpectedly terminated), weak references may have been corrupted in memory but may not yet have been corrupted in data repository 114. As described above, file system manager 108 can generate associated log entries indicating that weak references should be corrupted. Therefore, log entries can be replayed after the failure is resolved to ensure that weak references are corrupted or removed from data repository 114. Thus, weak references to blocks can be corrupted even if the blocks have not been changed (e.g., regardless of whether the associated modification operation was successfully completed before or during the failure), thereby ensuring that data corruption does not occur.

[0043] Additionally, it can be determined whether data repository 114 supports write persistence. Without write persistence, the logs themselves may not be stored correctly or intact for access after a failure is resolved. In cases where data repository 114 does not support write persistence, an unplanned shutdown or other unexpected failure has occurred, and / or it is determined that log entries are incorrect (e.g., log entries may not pass checksums), at least a portion of the deduplication hash table 112 can be cleared to avoid incorrectly deduplicating data in data repository 114. Returning to the example above, if a log entry indicates that a weak reference should be destroyed or removed, the weak reference state can be updated accordingly, regardless of whether the block has been modified, thus ensuring that the block is not incorrectly used for deduplication. In such cases, the deduplication engine 110 can correspondingly regenerate at least a portion of the deduplication hash table 112.

[0044] Figure 2A An overview of an example method 200 for processing file system blocks based on weak references to allocated logical clusters is shown. In the example, aspects of method 200 are determined by the storage system (such as...) Figure 1 The storage system 102 in the middle) performs this. For example, a deduplication engine (such as deduplication engine 110) can perform such an aspect. Method 200 can iterate through blocks to generate or update the deduplication hash table, such as Figure 1 The deduplication hash table 112 in the middle.

[0045] Deduplication engines can utilize file system managers (e.g., Figure 1 The API of the file system manager (108) is a user-mode service. In some other examples, such aspects do not need to be implemented as user-mode services, but can be implemented in kernel mode, or at least some of such functionality can be integrated into the file system manager. Similarly, the aspects described herein with respect to the file system manager can alternatively be implemented as part of a user-mode service, etc.

[0046] Method 200 begins with operation 202, where the modified file system block is identified. This block may have been modified since a previous execution of method 200. In the example, the file system manager (such as...) Figure 1 The file system manager 108 in the file system provides indications about one or more changed blocks, or, as another example, snapshots are evaluated to identify changes that have occurred since the snapshot was taken. Therefore, it will be understood that any of a variety of techniques can be used to identify changed file system blocks.

[0047] In a 204 error, the determination checks whether the block has a broken weak reference associated with it. In one example, this determination includes evaluating the reference count associated with the block. In other examples, the determination includes using APIs associated with the file system manager to determine the weak reference state for the block (e.g., no weak reference, intact weak reference, or broken weak reference, etc.).

[0048] If it is determined that a weak reference to the corrupted block exists, the process branch "Yes" proceeds to operation 206, where the entry associated with the block is retrieved from a deduplication hash table (e.g., ...). Figure 1 The deduplication hash table 112 in the middle is removed. In some cases, the hash table provides a mapping between block locations and hashes, allowing entries to be identified using the corresponding block locations. The process proceeds to operation 208, which will be discussed below.

[0049] However, if it is determined that the block does not have a corrupted weak reference (e.g., the block has no associated weak references), the flow alternatively branches "No" to operation 208. Operations 202 through 206 are shown with dashed lines to indicate that in some examples such aspects may not be performed. For example, if the implementation of the exposed weak reference functionality does not provide a corrupted state and instead provides an intact weak reference state and a no-weak-reference state, then operations 204 through 206 can be omitted. In such a case, the deduplicated hash table can be processed periodically to remove stale hashes. As another example, operation 202 can be omitted by processing all blocks or a subset of available blocks, regardless of whether they have been modified.

[0050] The process arrives at, or, as mentioned above, in some other examples may begin with operation 208, where a weak reference to a file system block is generated. Weak references can be used with file system managers (such as...) Figure 1 The API calls of the file system manager (108) in the file system are used to generate the reference count, thereby updating the reference count associated with the block to reflect the weak reference accordingly. While the example is described in terms of updating the reference count (which, in some examples, can also be used to reflect one or more strong references to the block), it will be understood that any of a variety of other techniques can be used. For example, the block may have an associated weak reference attribute, or a separate data structure may be used to store entries indicating weak references to the file system block.

[0051] The process proceeds to operation 210, where a hash is generated based on data stored in file system blocks. It will be understood that any of a variety of techniques can be used to hash the block data, including but not limited to SHA-1, SHA-2, or MD5. In other examples, pre-existing hashes associated with blocks can be used. For instance, the file system can perform checksums on blocks to ensure data integrity, and such checksums can, in some cases, be further used as hashes of the block data.

[0052] In determining 212, determining whether the hash exists in the hash table, such as... Figure 1 The deduplicated hash table 112 is used in the process. The keys of the hash table can be block hashes, such that the determination involves attempting to retrieve a value from the hash table using the block hash generated at operation 210. If no value is returned, it can be determined that the hash does not exist in the hash table.

[0053] Therefore, if it is determined that the hash does not exist in the hash table, the process branch "No" to operation 214, where an association is generated between the block hash and the file system block. In the example, the hash is used as the key for the hash table entry, while the logical address or other identifier for the file system block is used as the value. It will be understood that any of a variety of other techniques can be used to generate entries in the deduplicated hash table. For example, if a handle is generated for the block as a result of generating a weak reference in operation 208, the handle can be stored in association with the generated hash. The process then returns to operation 202 (or, in some other examples, operation 204 or 208), allowing the file system's additional blocks to be processed according to method 200.

[0054] However, if instead it is determined at determination 212 that the hash exists in the hash table, the process branch "yes" to operation 216, where an existing block is identified using the hash table. For example, a value associated with the block hash in the hash table can identify or otherwise reference an existing block (e.g., using an address or handle). As described above, any of the various other techniques can be used to associate a block hash with a block, such that the association can be used to identify the block using the block hash.

[0055] In determination 217, it is determined (e.g., as identified in operation 202) whether the current block is the same block as an entry in the hash table (e.g., as determined in determination 212). For example, the hash table may have an entry for that block when an existing hash table is reused or when a block is indicated to have been changed more than once. Therefore, if it is determined that the current block is the same block as an entry in the hash table, the flow branch "Yes" proceeds to operation 202.

[0056] In contrast, if it is alternatively determined that the current block is not the same block as an entry in the hash table, the process alternatively branches "No" to determination 218, where it is determined whether an intact weak reference exists on the existing block. For example, the existing block may have been processed according to a previous iteration of method 200, such that a weak reference was generated according to aspects of operation 208. However, if the existing block has since been modified, the weak reference may have been corrupted or removed. Therefore, determination 218 may include using API calls to the file system manager to determine whether the weak reference is still intact. In some other examples, determination 218 includes accessing the reference counting data structure associated with the block to determine the weak reference state indicated by the reference counting data structure.

[0057] If, at determination 218, the weak reference is determined to be not intact (e.g., it has since been corrupted or no longer exists), the process branch "No" proceeds to operation 214, as discussed above. Therefore, if the block hash in the deduplication hash table may no longer accurately reflect the state of the data stored by that block, the existing block is not used to perform deduplication. Instead, the block currently being processed by method 200 (e.g., as hashed at operation 210) is added to the deduplication hash table so that it can be used to perform deduplication in the future if a different block with its hash is identified.

[0058] However, if it is determined that the weak reference on the existing block is intact, the process branch "yes" to operation 220, where an instruction to replace the block with the existing block is generated. In the example, this instruction is provided to the file system manager using an API call (thus...). Figure 2B The aspects of method 250 are performed (as described below). In some cases, determination 218 can be omitted, allowing the process to proceed directly from determination 217 to operation 220. For example, the file system manager can alternatively verify weak references on both existing and new blocks, as described below. It will be understood that while aspects of method 200 are described in relation to the API provided by the file system manager, similar techniques can be applied more directly to file system metadata. For example, operation 220 can alternatively include generating strong references to existing blocks and correspondingly using those strong references to update the file's metadata. Such aspects can be similar to... Figure 2B Those are methods 250. Similar to operation 214, the process can then return to operation 202 (or, in some other examples, operation 208), so that another block can be processed according to method 200.

[0059] Figure 2B An overview of an example method 250 for performing deduplication of file system blocks based on weak references to allocated logical clusters is shown. In the example, aspects of method 250 are determined by the storage system (such as...) Figure 1The storage system 102 in the middle) performs this function. For example, a file system manager (such as file system manager 108) can perform such a function. As mentioned above, the file system manager can provide the function executed by the storage system 102. Figure 2A Method 200 of various deduplication engines (e.g., Figure 1 The deduplication engine (110) uses the API. In some other examples, such aspects do not need to be implemented by the file system manager and such functionality can be integrated into the deduplication engine (e.g., in user mode and / or kernel mode).

[0060] Method 250 begins with operation 252, where an instruction is received to use an existing block instead of a new block. For example, this instruction could be used to execute the above-mentioned... Figure 2A The result of the aspect of operation 220 under discussion is received. In some cases, the instruction includes the address, handle, or other identifier for each block in the existing and new blocks.

[0061] The process proceeds to operation 253, where existing and new blocks are locked to ensure that data stored by either block remains unchanged during the deduplication operation. Although associated weak references indicate data during hash generation (e.g., Figure 2A The transition period between operation 210 of method 200 and identifying the block as a deduplication candidate (e.g., determining 212 and 217) remains unchanged, but operation 253 can be performed to ensure that the data of any block does not have a race condition by which it is changed when a strong reference to a new block is replaced by a strong reference to an existing block.

[0062] As mentioned above, locking on an existing or new block does not need to prevent modification operations on the file associated with it. For example, other blocks associated with the file can be modified normally without considering deduplication. Alternatively, modification to a normal block or an existing block can break a strong reference to the corresponding block and allow another block to be allocated accordingly (e.g., as part of a write-on-demand allocation operation). In some other examples, such modification operations can be alternatively trapped and temporarily suspended so that they are retried or resumed once deduplication is complete (e.g., as part of operation 266 discussed below).

[0063] In some other examples, operation 253 may alternatively include incrementing the strong reference count for each block in the existing and new blocks. As described above, a strong reference count of "2" or greater will cause the new block to be allocated in response to an associated modification operation (e.g., according to the write-on-demand technique described above), allowing the file associated with the block to be modified using the newly allocated block. Similarly, a strong reference count of "1" indicates that there are no other strong references to the block, making it impossible to modify the block, and the strong references also prevent the block from being deallocated. As an example of each block being associated with a file, the strong reference counts for both the existing and new blocks can be incremented from "1" to "2". Therefore, some examples of method 250 can reduce or even avoid the use of locking during deduplication.

[0064] Determining 254 involves determining whether a valid weak reference exists on an existing block. For example, if the existing block has been modified since the weak reference was created, the weak reference may have been broken or removed. Determining 254 may include accessing the reference count or other data structure associated with the existing block to determine the state of the weak reference associated with the existing block. While method 250 is shown as locking the existing block and the new block and subsequently verifying the associated weak reference, it will be understood that in some other examples, these aspects may serve as... Figure 2A This is performed as part of method 200. For example, determination 218 may also include locking an existing block as part of determining whether an associated weak reference is intact, and may also include locking a new block and also verifying its associated weak reference. Therefore, locking and weak reference verification can be performed by the file system manager, the deduplication engine, or any combination thereof.

[0065] If it is determined that a weak reference on an existing block is not intact (e.g., it has been corrupted or no longer exists), the process branch is "No" and ends at operation 256. Operation 256 may also include unlocking both the existing and new blocks, since no deduplication operation will be performed on any blocks. Similarly, in the example where the strong reference count is incremented at operation 253, the strong reference counts associated with the existing and new blocks can each be decremented accordingly (e.g., returning to "1" in the example above). Therefore, deduplication does not occur if the deduplication hash table may no longer accurately reflect the state of the data stored by the existing blocks.

[0066] However, if it is alternatively determined that the weak references on the existing block are intact, the flow alternatively branches "yes" to determination 258, where it determines whether the weak references on the new block are intact. Similar to determination 254, the data stored by the new block may have changed, making referencing the existing block instead of the new block cause data corruption. As an example, determination 258 may include accessing the reference count or other data structure associated with the new block to determine the state of the weak references associated with the new block.

[0067] If it is determined that the weak reference on the new block is not intact (e.g., it has been corrupted or no longer exists), the process branch "No" and ends at operation 256. Therefore, deduplication does not occur if the deduplication hash table may no longer accurately reflect the state of the data stored by the new block.

[0068] In contrast, if it is determined that the weak references on the new block are intact (and, as mentioned above, the weak references on the existing block are also intact), the flow branch "yes" to operation 260, where a strong reference to the existing block is generated. Aspects of operation 260 may include incrementing the reference count associated with the existing block to indicate that the existing block is now a shared block (e.g., the existing block will be referenced in place of the new block). Where the strong reference count was incremented in operation 253, the incremented strong reference count on the existing block can be preserved (e.g., at "2" in the example above) instead of being further incremented in operation 260. Therefore, if the existing block was previously used to represent a portion of the first file (thus having a reference count of "1" before method 250), the strong reference generated in operation 260 updates the reference count to indicate that the existing block is now further used to represent a portion of the second file (now having a reference count of "2"). As mentioned above, this indicates that the existing block is now a shared block, ensuring that the block is not modified due to operations associated with the first or second file that could potentially corrupt data in other files. Instead, the increased reference count ensures that changes to each associated file remain distinct.

[0069] In operation 262, the metadata associated with the file to which the new block is attached is updated using the strong reference generated at operation 260. For example, the metadata is updated so that instead of referencing the new block, it references an existing block instead. Operations 260 and 262 are shown with dashed box 270 to indicate that in some examples these aspects can be performed using the file system manager's block cloning capabilities, where existing blocks are used to represent data for multiple files. In cases where a modification operation is captured or suspended, instructions can be generated to retry the modification operation or to allow the modification operation to be resumed, etc.

[0070] The process proceeds to operation 264, where the reference count of the new block is decremented because the file previously associated with the new block no longer references it. Therefore, in some cases, the new block can be deleted or otherwise released to store additional data, such as if the new block is only referenced by the aforementioned file (making the updated reference count "0"). In the example above, where operation 253 involves incrementing the associated strong reference count, operation 264 can produce a strong reference count for the new block decreasing from "2" to "1". As another example where a lock is used at operation 253 and the new block is referenced by a single file, the reference count can decrease from "1" to "0". In other cases, the new block can be retained, such as if the updated reference count remains non-zero.

[0071] In operation 266, both the existing and new blocks are unlocked. For example, if the new block has a reference count of "1", it can be modified accordingly. As another example, if the new block has a reference count of "0", it can be overwritten. In yet another example, an existing block can have a reference count of "1", making it possible to modify it, for example, when a modification operation on an associated file occurs during deduplication. Therefore, the file can instead reference the newly allocated block instead of the existing one.

[0072] If the strong reference count is incremented in operation 253, operation 266 may include further decrementing the strong reference count associated with the new block. As mentioned above, the incrementing strong reference count of the existing block is preserved at operation 260 because the existing block is now referenced instead of the new block. However, the strong reference count of the new block is first decremented at operation 264 to indicate that the new block is no longer referenced by the associated file, such that the strong reference count is further decremented at operation 266 to take into account the increment performed at operation 253. Thus, returning to the example above, the strong reference count for the new block is decremented from "2" to "1" at operation 264 and further decremented to "0" at operation 266. As mentioned above, captured or temporarily suspended modification operations can be retried or resumed as part of operation 266. The process terminates at operation 266.

[0073] Figure 3 An overview of an example method 300 for writing to a file system block is shown, according to various aspects of this disclosure. In the example, aspects of method 300 are determined by a storage system (such as...) Figure 1The execution occurs in storage system 102. For example, a file system manager (such as file system manager 108) can execute such an aspect. Method 300 begins at operation 302, where data to be written to the block is received. In the example, the address where data is to be written is also received, as can be the case when performing an in-situ modification of data stored by a given block. In some other examples, the block is determined from a set of available or otherwise unallocated blocks.

[0074] The process proceeds to determination 304, where it is determined whether the block has a weak reference. Determination 304 may include accessing the reference count or other data structure associated with the block to determine the weak reference state associated with the existing block. For example, the three most significant bits of the reference count can be accessed and evaluated to determine whether a healthy weak reference state exists associated with the block.

[0075] If it is determined that the block has an intact weak reference, the flow branch "yes" to operation 306, where the metadata is updated to indicate that the weak reference has been destroyed. Returning to the example above, the three most significant bits of the reference count can be updated to indicate the destroyed weak reference state. In other examples, the attributes associated with the block can be updated accordingly. Therefore, it will be understood that any of a variety of techniques can be used to determine whether the block has an associated weak reference and to update the weak reference state accordingly. In some other examples, the destroyed reference state may not be used, allowing the intact weak reference to be removed instead at operation 306. As another example, the intact weak reference state can be indicated using at least a portion of a data structure that is inadvertently recovered as part of a modification operation, allowing the weak reference to be automatically cleared as a result of the modification operation (e.g., the operation lacking method 300). In some cases, operations 304 and 306 may also include locking the block while destroying the weak reference to ensure that another operation did not determine the weak reference was intact before operation 306 and subsequently accessed the block after it was modified by operation 308. As another example, compare and swap operations can be used to automatically update the metadata.

[0076] The process proceeds to operation 308, where data is written to the block. In the example, writing data to the block and updating metadata at operation 306 may not occur simultaneously, meaning that for at least some time, the system may be in a state where the data stored in the block has changed, but the metadata still indicates a weak reference. Therefore, logging techniques can be used to indicate when blocks are written to mitigate file system inconsistencies that may be caused by intervention failures. The following is about... Figure 4 Method 400 provides additional discussion on these aspects. The process terminates at operation 308.

[0077] Returning to determination 304, if it is alternatively determined that the block does not have a weak reference, the process branch "No" to operation 308 discussed above. The process terminates at operation 308. It will be understood that in some other examples, aspects similar to method 300 can be performed in additional or alternative scenarios where the block is modified or not modified. For example, an operation associated with changing the strong reference count (e.g., similar to operation 302) can cause a weak reference to be destroyed or removed (e.g., similar to operation 306) and the file system metadata can be updated to reflect the changed strong reference count accordingly (e.g., similar to operation 308). Similar techniques can be applied to situations where an operation reduces the strong reference count, or more specifically, where an operation reduces the strong reference count to below "2" (e.g., the shared block is no longer shared or the block can be released). Furthermore, as mentioned above, destroying or removing a weak reference can be additionally or alternatively conditional on the modification of the block.

[0078] Figure 4 An overview of an example method 400 for restoring a file system state associated with weak references after a failure, according to aspects of this disclosure, is shown. In the example, aspects of method 400 are respectively executed... Figure 2A and 2B Methods 200 and 250 are executed before the fault occurs. For example, method 400 can be executed after the fault has occurred.

[0079] As described above, block modifications and associated weak reference updates may not occur simultaneously (e.g., be written to storage), making intervention failures potentially lead to inconsistent states. In such cases, if a new block replaces the previously overwritten block using the deduplication aspects described herein, relying on now incorrect weak references could cause data corruption.

[0080] Thus, method 400 begins with determination 402, where it is determined whether write persistence is enabled by the data repository (e.g., Figure 1 The data repository (114) in the process supports this. For example, determining this may include evaluating one or more capabilities and / or hardware characteristics of the data repository to determine whether the data repository can commit changes in the event of a failure. If it is determined that the data repository does not support write persistence, the process branch "No" to operation 406, where the deduplicated hash table (e.g., Figure 1 At least a portion of the deduplication hash table 112 in the system can be cleared to avoid incorrect deduplication. Therefore, if it cannot be determined whether the file system operations prior to the failure were correctly logged, at least a portion of the hash table can be cleared, allowing the deduplication engine (e.g., ...) to... Figure 1 The deduplication engine 112 in the process will not unintentionally cause data corruption. The process terminates at operation 406.

[0081] If it is alternatively determined that the data repository supports write persistence, the process branch "Yes" to operation 404, where the file system log is accessed. In the example, the file system manager (e.g., Figure 1 The file system manager 108 maintains a log of operations that can be replayed in the event of a failure. For example, this log may include information about block execution as described above. Figure 3 The instructions for operation 308 (and in some cases, operation 306) of method 300 are discussed.

[0082] The process proceeds to determination 408, where it is determined whether the log entry has been verified. In the example, verification includes generating a checksum for the entry and comparing the generated checksum with a stored checksum to determine if the log entry is corrupted. If the log entry has not been verified, the process branches "No" to operation 406, where at least a portion of the deduplication hash table is cleared as described above. The process terminates at operation 406.

[0083] In contrast, if the log entry is verified at checkpoint 408, the process alternatively branches to "Yes" at operation 410, where the log entry is replayed, thereby updating the weak reference associated with the block accordingly. In some cases, it may be impossible to determine whether a write to a block was successful, making it correct to retain an intact weak reference on the block if the write failed, but incorrect if the write succeeded. Therefore, replaying the log entry to destroy or otherwise remove the associated weak reference ensures that the block is not incorrectly used for deduplication, as might be the case when a write operation succeeds. Furthermore, the deduplication hash table does not need to be cleared, and the entries in the deduplication hash table accurately reflect the state of the file system blocks. The process terminates at operation 410.

[0084] Figures 5 to 8 The associated description provides a discussion of various operating environments in which the aspects of this disclosure can be practiced. However, regarding Figures 5 to 8 The devices and systems shown and discussed are for illustrative purposes only and are not intended to limit the wide range of computing device configurations that can be used to practice the aspects of this disclosure described herein.

[0085] Figure 5 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 500 that can be used to practice aspects of the present invention. The computing device components described below are applicable to the computing device described above, including... Figure 1The storage system 102 and computing device 104 are included. In a basic configuration, the computing device 500 may include at least one processing unit 502 and system memory 504. Depending on the configuration and type of the computing device, the system memory 504 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories.

[0086] System memory 504 may include an operating system 505 and one or more program modules 506 suitable for running software applications 520, such as one or more components supported by the system described herein. As an example, system memory 504 may store a file system manager 524 and a deduplication engine 526. The operating system 505 may, for example, be suitable for controlling the operation of computing device 500.

[0087] Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is... Figure 5 The components within the dashed line 508 are shown. The computing device 500 may have additional features or functions. For example, the computing device 500 may also include additional data storage devices (removable and / or non-removable), such as, for example, a disk, optical disc, or magnetic tape. Such additional storage... Figure 5 The image shows a removable storage device 509 and a non-removable storage device 510.

[0088] As described above, multiple program modules and data files can be stored in system memory 504. While executing on processing unit 502, program module 506 (e.g., application 520) can perform processes including, but not limited to, those described herein. Other program modules that can be used according to aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, PowerPoint presentation applications, drawing or computer-aided applications, etc.

[0089] Furthermore, embodiments of this disclosure can be practiced in areas including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or circuits on a single chip containing electronic components or microprocessors. For example, embodiments of this disclosure can be practiced via a system-on-a-chip (SoC), wherein... Figure 5Each or many of the components shown are mounted on a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the functionality described herein regarding the ability to switch client protocols can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 500. Embodiments of this disclosure may also be practiced using other techniques capable of performing logical operations, such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure may be practiced in general-purpose computers or any other circuit or system.

[0090] The computing device 500 may also have one or more input devices 512, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Multiple output devices 514 (such as a monitor, speaker, printer, etc.) may also be included. The above devices are examples and other devices may be used. The computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 550. Examples of suitable communication connections 516 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial buses (USB), parallel and / or serial ports.

[0091] The term "computer-readable medium" as used herein can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 504, removable storage device 509, and non-removable storage device 510 are examples of computer storage media (e.g., memory storage devices). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and can be accessed by computing device 500. Any such computer storage medium can be part of computing device 500. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0092] Communication media can be embodied in computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, radio frequency (RF), infrared, and other wireless media).

[0093] Figure 6A and 6B A mobile computing device 600, such as a mobile phone, smartphone, wearable computer (e.g., a smartwatch), tablet computer, laptop computer, etc., is shown that can be used to practice embodiments of the present disclosure. In some aspects, the client may be a mobile computing device. References Figure 6A This illustrates one aspect of a mobile computing device 600 used to implement these aspects. In a basic configuration, the mobile computing device 600 is a handheld computer with both input and output elements. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610, which allow the user to input information into the mobile computing device 600. The display 605 of the mobile computing device 600 can also be used as an input device (e.g., a touchscreen display).

[0094] If included, the optional side input element 615 allows for additional user input. The side input element 615 can be a rotary switch, a button, or any other type of manual input element. Alternatively, the mobile computing device 600 can incorporate more or fewer input elements. For example, in some embodiments, the display 605 may not be a touchscreen.

[0095] In yet another alternative embodiment, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 may also include an optional keypad 635. The optional keypad 635 may be a physical keypad or a “soft” keypad generated on a touchscreen display.

[0096] In various embodiments, output elements include a display 605 for displaying a graphical user interface (GUI), a visual indicator 620 (e.g., a light-emitting diode), and / or an audio transducer 625 (e.g., a speaker). In some aspects, the mobile computing device 600 incorporates a vibration transducer for providing haptic feedback to a user. In another aspect, the mobile computing device 600 incorporates input and / or output ports, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0097] Figure 6B This is a block diagram illustrating one aspect of the architecture of a mobile computing device. That is, the mobile computing device 600 can be incorporated into system (e.g., architecture) 602 to implement certain aspects. In one embodiment, system 602 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 602 is integrated as a computing device, such as integrating a personal digital assistant (PDA) and a wireless phone.

[0098] One or more applications 666 may be loaded into memory 662 and run on or associated with operating system (OS) 664. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, etc. System 602 also includes a non-volatile storage area 668 within memory 662. The non-volatile storage area 668 may be used to store persistent information that should not be lost in the event of a power outage of system 602. Applications 666 may use and store information, such as emails or other messages used by email applications, in the non-volatile storage area 668. A synchronization application (not shown) also resides on system 602 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 668 synchronized with the corresponding information stored on the host computer. It should be understood that other applications may be loaded into memory 662 and run on the mobile computing device 600 described herein (e.g., search engines, extractor modules, relevance ranking modules, answer scoring modules, etc.).

[0099] System 602 has a power supply 670, which can be implemented as one or more batteries. The power supply 670 may also include an external power source, such as an AC adapter or a power docking station for replenishing or recharging the batteries.

[0100] System 602 may also include a radio interface layer 672, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 672 facilitates wireless connections between system 602 and the "external world" via a communications operator or service provider. Transmissions to and from radio interface layer 672 are conducted under the control of operating system 664. In other words, communications received by radio interface layer 672 can be propagated to application 666 via operating system 664, and vice versa.

[0101] A visual indicator 620 may be used to provide visual notifications, and / or an audio interface 674 may be used to generate auditory notifications via an audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light-emitting diode (LED) and the audio transducer 625 is a speaker. These devices may be directly coupled to a power supply 670 such that when activated, they remain on for a duration specified by the notification mechanism, even if the processor 660 and other components may be turned off to conserve battery power. The LED may be programmed to remain on indefinitely until the user takes an action to indicate the power-on status of the device. The audio interface 674 is used to provide and receive audible signals to and from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 may also be coupled to a microphone to receive audible input, such as to facilitate telephone conversations. According to embodiments of this disclosure, the microphone may also be used as an audio sensor to facilitate control of notifications, as described below. System 602 may also include a video interface 676, which enables the operation of the onboard camera 630 to record still images, video streams, etc.

[0102] The mobile computing device 600 implementing system 602 may have additional features or functions. For example, the mobile computing device 600 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage... Figure 6B The non-volatile storage region 668 is shown in the middle.

[0103] As described above, data / information generated or captured by mobile computing device 600 and stored via system 602 can be stored locally on mobile computing device 600, or the data can be stored on any number of storage media that can be accessed by the device via radio interface layer 672 or via a wired connection between mobile computing device 600 and a separate computing device associated with mobile computing device 600 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via mobile computing device 600 via radio interface layer 672 or via a distributed computing network. Similarly, according to well-known data / information transmission and storage methods (including email and collaborative data / information sharing systems), such data / information can be easily transferred between computing devices for storage and use.

[0104] Figure 7One aspect of the architecture of a system for processing data received at a computing system from a remote source (such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708), as described above, is illustrated. The content displayed at server device 702 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 722, a web portal 724, an email service 726, an instant messaging repository 728, or a social networking site 730.

[0105] The deduplication engine 720 can be employed by a client communicating with server device 702, and / or file system manager 721 can be employed by server device 702. Server device 702 can provide data to and from client computing devices (such as personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone)) via network 715. As an example, the aforementioned computer system can be embodied in personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone). In addition to receiving graphics data that can be preprocessed at the graphics generation system or post-processed at the receiving computing system, any of these embodiments of the computing device can obtain content from repository 716.

[0106] Figure 8 An exemplary tablet computing device 800 is shown that can perform one or more aspects of the present invention. Additionally, the aspects and functions described herein can operate on a distributed system (e.g., a cloud-based computing system) where application functions, memory, data storage and retrieval devices, and various processing functions can operate remotely to each other via a distributed computing network (such as the Internet or an intranet). Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall on which various types of user interfaces and information are projected. Interaction with a wide range of computing systems on which embodiments of the invention can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) capabilities for capturing and interpreting user gestures used to control functions of the computing device, etc.

[0107] As can be understood from the foregoing disclosure, one aspect of this technology relates to a system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: generating a weak reference on a first block of a file system; generating an entry in a deduplication hash table for the first block, the entry including a block hash and a block position; identifying a second block having a block hash that matches the block hash in the deduplication hash table for the first block; determining whether the weak reference on the first block is intact; and based on the determination that the weak reference on the first block is intact: generating a strong reference to the first block based on the block position of the entry in the deduplication hash table; and storing the strong reference to the first block as metadata for a file associated with the second block. In one example, generating a weak reference on the first block includes updating a reference counting data structure associated with the first block to indicate an intact weak reference state; and determining whether the weak reference on the first block is intact includes accessing the reference counting data structure. In another example, generating a weak reference on the first block includes providing an instruction to the file system manager to generate a weak reference on the first block; and determining whether the weak reference on the first block is intact includes requesting the weak reference state associated with the first block from the file system manager. In yet another example, the set of operations further includes storing a strong reference to the second block as metadata for the file based on the determination that the weak reference on the first block is not intact. In yet another example, the reference counting data structure includes a first sub-part associated with an intact weak reference state; and a second sub-part associated with a strong reference count. In yet another example, the set of operations further includes receiving an instruction to modify the first block; and, in response to the instruction, destroying the weak reference on the first block. In yet another example, determining whether the weak reference on the first block is intact also includes determining whether the weak reference on the second block is intact, and the set of operations further includes storing a strong reference to the second block as metadata for the file based on the determination that at least one of the weak references on the first block or the weak references on the second block are not intact. In another example, the set of operations also includes: determining that the first block is associated with a broken weak reference before generating a weak reference on the first block; and removing stale entries associated with the first block from the deduplication hash table based on the determination that the first block is associated with a broken weak reference.

[0108] On the other hand, this technology relates to a method for deduplicating data in a deduplicated file system. The method includes: generating a weak reference on a first block of the file system, wherein the weak reference is in a healthy weak reference state; generating an entry in a deduplication hash table for the first block; identifying a second block having a block hash that matches an entry in the deduplication hash table, wherein the second block is associated with a file in the file system; determining whether the weak reference on the first block is in a healthy weak reference state; and using the second block to represent a file in the file system based on the determination that the weak reference is not in a healthy weak reference state. In one example, the method further includes: removing an entry for the first block from the deduplication hash table based on the determination that the weak reference is in a corrupted weak reference state. In another example, generating a weak reference on the first block includes updating a reference counting data structure associated with the first block to indicate a healthy weak reference state; and determining whether the weak reference on the first block is healthy includes accessing the reference counting data structure. In yet another example, generating a weak reference on the first block includes providing an instruction to a file system manager to generate a weak reference on the first block; and determining whether the weak reference on the first block is healthy includes requesting the weak reference state associated with the first block from the file system manager.

[0109] In another aspect, this technology relates to a method for deduplicating data in a file system. The method includes: determining that a first weak reference on a first block of the file system is in a corrupted weak reference state; removing a first entry associated with the first block from a deduplication hash table based on the determination that the first weak reference is in a corrupted weak reference state; generating a second weak reference on the first block, wherein the second weak reference is in a healthy weak reference state; generating a second entry in the deduplication hash table for the first block; identifying a second block having a block hash that matches the second entry in the deduplication hash table; locking the first and second blocks of the file system; determining whether the first and second weak references are in a healthy weak reference state; using the first block instead of the second block to represent a file in the file system based on the determination that the first and second weak references are in a healthy weak reference state; and unlocking the first and second blocks of the file system. In one example, using the first block instead of the second block to represent a file includes: generating a strong reference to the first block based on the second entry in the deduplication hash table; and storing the strong reference to the first block as metadata for the file in the file system. In another example, generating a second weak reference on the first block includes updating the reference count data structure associated with the first block to indicate a healthy weak reference state; and determining whether the second weak reference is in a healthy weak reference state includes accessing the reference count data structure. In yet another example, using the first block instead of the second block to represent a file includes providing an instruction to the file system manager to use the first block instead of the second block to represent the file. In yet another example, generating a second weak reference on the first block includes providing an instruction to the file system manager to generate a weak reference on the first block; and determining whether the second weak reference is in a healthy weak reference state includes requesting the weak reference state associated with the first block from the file system manager. In yet another example, the method further includes: receiving a modification operation on the file associated with the first block while the first block is locked; and allocating a third block associated with the file for the modification operation. In another example, the reference count data structure includes: a first sub-part associated with a weak reference state; and a second sub-part associated with a strong reference count. In yet another example, the first sub-part indicates a weak reference state selected from the group consisting of: a healthy weak reference state; a corrupted weak reference state; and a state with no weak references.

[0110] For example, aspects of this disclosure have been described above with reference to block diagrams and / or operating instructions of methods, systems, and computer program products according to aspects of this disclosure. The functions / actions noted in the boxes may occur in the order shown in any flowchart. For example, two boxes shown in a sequential manner may actually be executed substantially concurrently, or these boxes may sometimes be executed in reverse order, depending on the functions / actions involved.

[0111] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed disclosure as the best mode. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Whether shown and described in combination or separately, various features (both structural and methodological) are intended to be selectively included or omitted to produce embodiments with a particular set of features. Having provided the descriptions and illustrations of this application, those skilled in the art can conceive of variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application, without departing from the broader scope of the claimed disclosure.

Claims

1. A system comprising: At least one processor; as well as Memory, storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations including: A weak reference is generated on the first block of the file system, wherein the weak reference stores the association between the block location of the first block and the state of the first block after the weak reference is generated; For the first block, generate an entry in a deduplication hash table, the entry including the block hash and the block position; Identify a second block, the second block having a block hash that matches the block hash of the first block in the deduplication hash table; Determine whether the weak reference on the first block is intact; and Based on the determination that the weak reference on the first block is intact, it is indicated that the data in the first block has not been modified: Based on the block position of the entry in the deduplication hash table, a strong reference to the first block is generated; and The strong reference to the first block is stored as metadata for the second file associated with the second block.

2. The system according to claim 1, wherein: Generating the weak reference on the first block includes updating the reference count data structure associated with the first block to indicate a healthy weak reference state; and Determining whether the weak reference on the first block is intact includes accessing the reference count data structure.

3. The system according to claim 1, wherein: Generating the weak reference on the first block includes providing an instruction to the file system manager to generate the weak reference on the first block; and Determining whether the weak reference on the first block is intact includes requesting the weak reference state associated with the first block from the file system manager.

4. The system according to claim 1, wherein the set of operations further includes: Based on the determination that the weak reference on the first block is not intact, thereby indicating that the data in the first block has been modified, a strong reference to the second block is stored as metadata for the file.

5. The system according to claim 2, wherein the reference count data structure comprises: The first sub-part associated with the intact weak reference state; as well as The second sub-part associated with strong reference counting.

6. The system of claim 4, wherein the set of operations further comprises: Receive an instruction for the intention to modify the first block; as well as In response to the instruction intended to modify the first block, the weak reference on the first block is destroyed.

7. The system of claim 1, wherein determining whether the weak reference on the first block is intact further includes determining whether the weak reference on the second block is intact, and wherein the set of operations further includes: Based on determining that at least one of the weak references on the first block are not intact or the weak references on the second block are not intact, a strong reference to the second block is stored as metadata for the second file.

8. The system of claim 1, wherein the set of operations further comprises: Before the weak reference is generated on the first block, it is determined that the first block is associated with the broken weak reference; as well as Based on the determination that the first block is associated with the broken weak reference, the stale entry associated with the first block is removed from the deduplication hash table.

9. A method for deduplicating data in a file system, the method comprising: A weak reference is generated on the first block of the file system, wherein the weak reference is in an intact weak reference state, and wherein the weak reference is an entry in a data structure that stores the block location of the first block and the state of the first block after the weak reference is generated. Generate a deduplication hash table entry for the first block; Identify a second block, the second block having a block hash that matches the entry in the deduplication hash table, wherein the second block is associated with a file in the file system; Determine whether the weak reference on the first block is in the intact weak reference state, and the fact that the weak reference is in the intact weak reference state indicates that the data in the first block has not been modified; as well as Based on the determination that the weak reference is not in the intact weak reference state, the second block is used to represent the file in the file system; Identify a third block, the third block having a block hash that matches the block hash for the second block; Based on the determination that the weak reference to the second block is intact, the strong reference to the second block is stored as metadata for the file that includes the third block.

10. The method of claim 9, further comprising: Based on the determination that the weak reference is in a broken weak reference state, the entry for the first block is removed from the deduplication hash table.

11. The method of claim 9, wherein: Generating the weak reference on the first block includes updating the reference count data structure associated with the first block to indicate a healthy weak reference state; and Determining whether the weak reference on the first block is intact includes accessing the reference count data structure.

12. The method of claim 9, wherein: Generating the weak reference on the first block includes providing an instruction to the file system manager to generate the weak reference on the first block; and Determining whether the weak reference on the first block is intact includes requesting the weak reference state associated with the first block from the file system manager.

13. A method for deduplicating data in a file system, the method comprising: Determine that the first weak reference on the first block of the file system is in a broken weak reference state; Based on the determination that the first weak reference is in a broken weak reference state, the first entry associated with the first block is removed from the deduplication hash table; Generate a second weak reference on the first block, wherein the second weak reference is in an intact weak reference state; A second entry is generated in the deduplication hash table for the first block, wherein the second entry stores the second weak reference and the deduplication hash table stores the association between the block position of the first block and the state of the first block after the second weak reference is generated; Identify the second block, which has a block hash that matches the second entry in the deduplication hash table; Lock the first and second blocks of the file system; Determine whether the first weak reference and the second weak reference are in the intact weak reference state; Based on the determination that the first weak reference and the second weak reference are in the intact weak reference state, thereby indicating that the data of the first block of the first weak reference has not been modified, the first block is used instead of the second block to represent the file of the file system, and a strong reference to the first block is stored as metadata for the second file associated with the second block; and Unlock the first and second blocks of the file system.

14. The method of claim 13, wherein: Generating the second weak reference on the first block includes updating the reference count data structure associated with the first block to indicate the intact weak reference state; and Determining whether the second weak reference is in the intact weak reference state includes accessing the reference counting data structure.

15. The method of claim 13, wherein using the first block instead of the second block to represent the file includes providing an instruction to a file system manager to use the first block instead of the second block to represent the file.

16. The method of claim 13, wherein: Generating the second weak reference on the first block includes providing an instruction to the file system manager to generate the weak reference on the first block; and Determining whether the second weak reference is in the intact weak reference state includes requesting the weak reference state associated with the first block from the file system manager.

17. The method of claim 13, further comprising: While the first block is locked, modification operations are received on the file associated with the first block; as well as A third block associated with the file is assigned for the modification operation.

18. The method of claim 14, wherein the reference count data structure comprises: The first sub-part associated with the weak reference state; as well as The second sub-part associated with strong reference counting.

19. The method of claim 18, wherein the first sub-part indicates a weak reference state selected from the group consisting of: The intact weak reference state; The broken weak reference state; and No weak references.

Images