Data management method and device

Abstract

The invention discloses a data management method and device, and belongs to the technical field of file systems. The method is applied to a file system layer. The method comprises the following steps: acquiring a first data block in a first file to be written into an external memory; writing the first data block into an external memory, and adding a first non-deduplication mark for the first file; and based on the first non-deduplication mark, performing an offline deduplication operation on the first data block. According to the method, the file system layer adds the non-deduplication mark for the file, and executes the offline deduplication operation on the data block based on the non-deduplication mark, so that incremental deduplication of the data can be realized, and the time overhead and the calculation overhead of data deduplication are reduced.

Description

Technical Field

[0001] The present application relates to the technical field of file systems, and in particular to a data management method and device thereof. Background Art

[0002] Deduplication technology is also called de-duplication technology. Its principle is to reduce redundant data by retaining only one physical block copy for logical data blocks with the same data and indexing the access to these logical blocks to this physical block. Deduplication technology can not only reduce real-time redundant writes on the critical path and extend the service life of the equipment, but also save storage capacity and realize efficient reuse of storage space.

[0003] Current data deduplication generally divides files into blocks, then compares the blocks based on their fingerprint information, and references logical files with the same logical data content to a unique shared physical copy to eliminate redundant data.

[0004] However, there is still room for performance improvement in current data deduplication technology. Summary of the invention

[0005] The present application provides a data management method and device. The file system layer of the present application adds a non-deduplicated mark to the file, and performs offline deduplication operations on the data block based on the non-deduplicated mark, which can realize incremental deduplication of data and reduce the time and computing overhead of data deduplication. The technical solution provided by the present application is as follows:

[0006] In a first aspect, the present application provides a data management method. The method is applied to the file system layer. The method comprises: obtaining a first data block in a first file to be written to an external memory; writing the first data block to the external memory and adding a first non-deduplicated mark to the first file; and performing an offline deduplication operation on the first data block based on the first non-deduplicated mark.

[0007] In this data management method, the file system layer adds a non-deduplicated mark to the file, and performs offline deduplication operations on the data block based on the non-deduplicated mark, which can achieve incremental deduplication of data, reduce the time and computing overhead of data deduplication, and can also reduce the space overhead caused by incremental data recording in traditional solutions, saving storage space to a large extent. At the same time, since the file system is a user-visible granularity, by adding a non-deduplicated mark to the file, it provides conditions for the user to specify the scope of offline deduplication, which helps to provide users with interactive functions such as file semantic perception and deduplication range definition, so as to meet the personalized needs of users.

[0008] Optionally, before the file system layer writes each data block to the external memory, it is necessary to determine whether the data block is suitable for online deduplication operation. For multiple data blocks in the file, some of the multiple data blocks may be suitable for online deduplication operation, while some of the data blocks are not suitable for online deduplication operation. Therefore, when the first data block is not suitable for online deduplication operation, the file system layer can also add a second non-deduplicated mark to the first data block to mark that the first data block needs to perform offline deduplication operation. The method also includes: the file system layer adds a second non-deduplicated mark to the first data block.

[0009] At this time, based on the first non-deduplication mark, an offline deduplication operation is performed on the first data block, including: based on the first non-deduplication mark, determining the first file from multiple files written to the external memory; based on the second non-deduplication mark, determining the first data block from multiple data blocks in the first file; performing an offline deduplication operation on the first data block.

[0010] When the user needs to specify the range of the deduplication operation to be performed on the first file, a deduplication instruction can be sent to the file system layer to indicate the range. For example, when the user instructs the first file to be stored on a computing device through the client used by the user, when the user needs to specify the range, the user can send a deduplication instruction to the computing device through the client used by the user, so that the file system layer of the computing device obtains information for performing a deduplication operation on the specified range of the first file through the deduplication instruction. The method also includes: receiving a deduplication instruction sent by the user, the deduplication instruction instructing to perform a deduplication operation on the specified range of the first file. Accordingly, based on the first non-deduplication mark, an offline deduplication operation is performed on the first data block, including: based on the deduplication instruction, determining a specified range in multiple files marked with the first non-deduplication mark, and performing an offline deduplication operation on the data blocks included in the specified range.

[0011] Optionally, the specified range is the first file, a file set to which the first file belongs, a directory of the first file, or a file cluster of the first file.

[0012] In one implementation, an offline deduplication operation is performed on a first data block based on a first non-deduplicated mark, including: determining the first data block based on the first non-deduplicated mark; matching the fingerprint information of the first data block with the fingerprint information stored in at least one of a memory and an external memory; when the fingerprint information of the first data block matches the stored fingerprint information, performing deduplication processing on the first data block; when the fingerprint information of the first data block does not match the stored fingerprint information, updating the key value of the first data block to the external memory based on the fingerprint information of the first data block.

[0013] Among them, since the memory has a faster access speed and the fingerprint information stored in the memory is more likely to be the fingerprint information of duplicate data, the file system layer can optionally prioritize matching the fingerprint information of the first data block with the fingerprint information stored in the memory. Then the fingerprint information of the first data block is matched with the fingerprint information stored in at least one of the memory and the external memory, including: matching the fingerprint information of the first data block with the fingerprint information stored in the memory; when the fingerprint information stored in the memory does not match the fingerprint information of the first data block, continue to match the fingerprint information of the first data block with the fingerprint information stored in the external memory.

[0014] In one implementation, the external memory includes a first-level storage unit and a second-level storage unit; the entries stored in the first-level storage unit are entries removed from the memory; the entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging.

[0015] When the external memory includes a first-level storage unit and a second-level storage unit, since the first-level storage unit and the second-level storage unit may both record key values, when matching the fingerprint information of the first data block with the fingerprint information stored in the external memory, the fingerprint information of the first data block may be optionally matched with the fingerprint information recorded in the first-level storage unit and the second-level storage unit. In another implementation, the offline deduplication operation is generally performed at a specified time, such as when the file system is idle, and before the specified time, the file system may optionally submit all key values ​​recorded in the first-level storage unit to the second-level storage unit. At this time, since the key value is not recorded in the first-level storage unit when performing the offline deduplication operation, when matching the fingerprint information of the first data block with the fingerprint information stored in the external memory, the fingerprint information of the first data block is actually matched with the fingerprint information recorded in the second-level storage unit.

[0016] Optionally, the method also includes: receiving a deletion instruction indicating deletion of the second data block; obtaining the storage location of the key value of the second data block; when the key value of the second data block is stored in the memory and not stored in the external memory, deleting the key value of the second data block in the memory; when the key value of the second data block is not stored in the memory but stored in the external memory, recording a deletion event indicating deletion of the key value of the second data block, and executing the deletion event at a specified time to delete the key value of the second data block in the external memory; when the key value of the second data block is stored in the memory and stored in the external memory, deleting the key value of the second data block in both the memory and the external memory.

[0017] As can be seen from the above, when data needs to be deleted, the file system layer can immediately release the data reference and reclaim the storage space. When the expired key value is stored in the external storage, the file system layer does not need to delete the expired key value from the external storage immediately, but records the deletion event indicating the deletion of the expired key value, and deletes the expired key value based on the deletion event at a specified time, releasing the space occupied by the expired key value, thus avoiding the high overhead caused by the need to immediately delete the expired key value.

[0018] Optionally, the method further includes: determining whether the first data block is suitable for performing an online deduplication operation. Then, writing the first data block to an external memory, and adding a first non-deduplication mark to the first file, including: when the first data block is not suitable for performing an online deduplication operation, writing the first data block to an external memory, and adding a first non-deduplication mark to the first file.

[0019] In a second aspect, the present application provides a data management device. The device is applied to the file system layer. The device includes: an acquisition module, used to acquire a first data block in a first file to be written to an external memory; a writing module, used to write the first data block to the external memory and add a first non-deduplicated mark to the first file; a deduplication module, used to perform an offline deduplication operation on the first data block based on the first non-deduplicated mark.

[0020] Optionally, the writing module is further used to add a second non-deduplicated mark to the first data block.

[0021] Optionally, the deduplication module is specifically used to: determine a first file from multiple files written to an external memory based on a first non-deduplication mark; determine a first data block from multiple data blocks in the first file based on a second non-deduplication mark; and perform offline deduplication operations on the first data block.

[0022] Optionally, the device also includes: an interactive module, used to receive a deduplication instruction sent by a user, the deduplication instruction instructing to perform a deduplication operation on a specified range of the first file; a deduplication module, specifically used to determine the specified range in multiple files marked with a first non-deduplication mark based on the deduplication instruction, and perform an offline deduplication operation on the data blocks included in the specified range.

[0023] Optionally, the specified range is the first file, a file set to which the first file belongs, a directory of the first file, or a file cluster of the first file.

[0024] Optionally, a deduplication module is specifically used to: determine a first data block based on a first non-deduplicated mark; match the fingerprint information of the first data block with the fingerprint information stored in at least one of the internal memory and the external memory; when the fingerprint information of the first data block matches the stored fingerprint information, perform deduplication processing on the first data block; when the fingerprint information of the first data block does not match the stored fingerprint information, update the key value of the first data block to the external memory based on the fingerprint information of the first data block.

[0025] Optionally, a deduplication module is specifically used to: match the fingerprint information of the first data block with the fingerprint information stored in the memory; when the fingerprint information stored in the memory does not match the fingerprint information of the first data block, continue to match the fingerprint information of the first data block with the fingerprint information stored in the external memory.

[0026] Optionally, the external memory includes a first-level storage unit and a second-level storage unit; the entries stored in the first-level storage unit are entries removed from the memory; the entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging.

[0027] Optionally, the device also includes: an interaction module, used to receive a deletion instruction indicating deletion of the second data block; an acquisition module, used to obtain the storage location of the key value of the second data block; a deletion module, used to delete the key value of the second data block in the memory when the key value of the second data block is stored in the memory and not stored in the external memory; the deletion module is also used to record a deletion event indicating deletion of the key value of the second data block when the key value of the second data block is not stored in the memory but stored in the external memory, and execute the deletion event at a specified time to delete the key value of the second data block in the external memory; the deletion module is also used to delete the key value of the second data block in the memory and the external memory when the key value of the second data block is stored in the memory and stored in the external memory.

[0028] Optionally, the device also includes: a judgment module, used to judge whether the first data block is suitable for performing online deduplication operations; a writing module, specifically used to write the first data block to an external memory and add a first non-deduplicated mark to the first file when the first data block is not suitable for performing online deduplication operations.

[0029] In a third aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method provided in the first aspect of the present application and any possible implementation thereof.

[0030] In a fourth aspect, the present application provides a computing device cluster, including multiple computing devices, the multiple computing devices including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors running the program instructions, so that the computing device cluster executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0032] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of an operating system I / O stack provided in an embodiment of the present application;

[0034] Figure 2 It is a schematic diagram of an implementation environment involved in a data management method provided in an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of an implementation environment involved in another data management method provided in an embodiment of the present application;

[0036] Figure 4 is a flow chart of a data deduplication process provided by an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of determining whether a first data block is duplicate data provided by an embodiment of the present application;

[0038] Figure 6 It is a schematic diagram of memory management fingerprint information provided by an embodiment of the present application;

[0039] Figure 7 is a schematic diagram of a queue to be offline provided in an embodiment of the present application;

[0040] Figure 8 is a schematic diagram of a bitmap of a file provided in an embodiment of the present application;

[0041] Fig. 9 It is a schematic diagram of a process for determining a data block that needs to perform an offline deduplication operation provided by an embodiment of the present application;

[0042] Fig.10 This is a schematic diagram of determining whether a first data block is duplicate data provided by an embodiment of the present application;

[0043] Fig.11 It is a schematic diagram of a three-layer storage structure of a key value provided in an embodiment of the present application, and a memory uses a memory bucket to manage key values;

[0044] Fig.12 is a flow chart of another deduplication process provided by an embodiment of the present application;

[0045] Fig.13 This is a flowchart of a process of deleting data based on user instructions provided by an embodiment of the present application;

[0046] Fig.14 is a flowchart of deleting a key value of a second data block provided by an embodiment of the present application;

[0047] Fig.15 It is a structural diagram of a data management device provided in an embodiment of the present application;

[0048] Fig.16 is a structural diagram of another data management device provided in an embodiment of the present application;

[0049] Fig.17 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0050] Fig.18 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] To facilitate understanding, the technology and background involved in the embodiments of the present application are first explained below.

[0053] The log-structured file system (LFS) has the features of out-of-place update and garbage collection. Compared with traditional jorunaling file systems (such as EXT and XFS) and copy-on-write file systems (such as BtrFS and ZFS), it has better random write and small-granularity transaction update performance.

[0054] The flash-friendly file system (F2FS) is a log-structured file system designed for flash memory devices. It is widely used in smart terminal scenarios such as mobile phones and high-speed servers. The flash-friendly file system organizes the storage area into a log structure, writes data such as user data, file mapping and directory index in the form of logs, and converts random writes of data and file structures into sequential writes, thereby improving the overall write performance of the file system and slowing down the wear of devices with limited life (such as flash memory). It also has an efficient file fragment recovery mechanism. Currently, F2FS has been widely used in smart terminal devices (such as mobile phones and smart TVs) and servers, and these devices also have a large amount of duplicate data from application caches and backups, user duplicate files, and system updates and snapshots. Among them, the file system manages the data of the file system in units such as blocks, segments, sections, zones and file systems. The granularity of blocks, segments, sections, zones and file systems increases in sequence.

[0055] Logical / virtual / physical address (LA / VA / PA) is the three-layer address used in this application. The logical address refers to the offset of the data block relative to the file, that is, the offset of the data block within the file. The physical address refers to the offset of the data block in the entire file system address space, that is, the actual location of the data in the physical storage space. The physical address is used to isolate the two to reduce the complexity of deduplication file system mapping.

[0056] The reference count (RC) is used to mark the number of times a copy is referenced in the deduplication system, that is, how many logical blocks reference a physical block.

[0057] Device mapper is a framework provided by the Linux kernel for mapping physical block devices to advanced virtual block devices. It is the basis of the logical volume manager.

[0058] The page cache is created by the operating system using memory and is an area used to cache data pages in external storage (such as a disk or solid-state drive).

[0059] Direct I / O refers to reading and writing external memory directly. The upper-level file system or application only records the file mapping or address index, rather than directly accessing the data content in the memory, without going through the page cache. Therefore, for data written through Direct I / O, additional reading is required to obtain the data content.

[0060] An inode is a data structure used to describe objects such as files and directories.

[0061] The super block is used to store the metadata of the file system. The super block contains information such as the file system type, size, status, and usage. Its function is to enable the operating system to identify and manage the file system.

[0062] A checkpoint is used to represent the consistent state of a system. In a file system, the consistent state of the system is saved to a persistent storage so that it can be restored based on the consistent state in the event of a failure.

[0063] Data deduplication technology is also called de-duplication technology. Its principle is to reduce redundant data by retaining only one physical block copy for logical data blocks with the same data and indexing the access to these logical blocks to this physical block. Data deduplication technology can not only reduce real-time redundant writes on the critical path and extend the service life of the equipment, but also save storage capacity and achieve efficient reuse of storage space. Current data deduplication generally divides files into data blocks, and then compares the data blocks according to the fingerprint information of the data blocks, and references the logical files with the same logical data content to the only shared physical copy to eliminate redundant data.

[0064] At present, according to the triggering timing, deduplication is generally divided into online deduplication and offline deduplication. Online deduplication is performed before the data is written to the disk, while offline deduplication is to retain the original data in the external memory first, and then trigger offline deduplication under specific conditions. The data management method provided in the embodiment of the present application mainly relates to online deduplication. The advantage of online deduplication is that since the deduplication process is completed before the data is written to the disk, the amount of data written can be reduced and redundant writing can be eliminated in real time. However, its disadvantage is that if the fingerprint (FP) information is stored in the memory, it is usually impossible to retain all fingerprint information, resulting in the inability to perform thorough deduplication. If it is stored in the external memory, since the external memory needs to be repeatedly read and written on the critical path of the data reading and writing process, it will significantly affect the input / output (I / O) speed of the system. This leads to the problems of high CPU overhead, long tail delay and high memory usage in the write path of online deduplication. Offline deduplication aims to write data to the storage device first, and then eliminate duplicate data when the system is idle, and it is impossible to obtain deduplication benefits immediately.

[0065] Deduplication can be done at different levels of the operating system I / O stack. Figure 1 Schematic diagram of an operating system I / O stack provided in an embodiment of the present application. Figure 1As shown, the operating system I / O stack includes: a virtual file system (VFS) layer, a file system (FS) layer, a block layer, and a hard disk driver layer. In some implementations, the hard disk driver layer can be a small computer system interface (SCSI) layer. Among them, the virtual file system is responsible for the general file abstract semantics, abstracts a unified operation interface for the upper layer, and switches different file systems on the I / O path. The file system is responsible for abstracting the concept of "file" and maintaining the location mapping of "file" data to the block layer. The block layer is used to make a unified abstraction of the underlying hardware devices, and is mainly responsible for the implementation of the I / O scheduling strategy. For example, the block layer collects batch I / Os based on the scheduling strategy and aggregates and sends them down, merges I / O requests to reduce the number of I / Os, etc. The hard disk driver layer is responsible for docking with the hardware disk.

[0066] At present, file systems have widely supported data deduplication technology (such as BtrFS, XFS, ZFS, DDFS, LessFS, etc.), which perform deduplication on online or offline paths respectively to achieve data sharing. At the same time, they use file system structures, such as file mapping, data checksum values, etc. to reduce the space overhead and performance impact brought by deduplication indexing, and combine deduplication technology with characteristics such as file semantics, data popularity and spatial locality to improve user interactivity.

[0067] DMdedup is a typical example of deduplication at the block layer. DMdedup uses the device mapper technology in the kernel to map two devices into a logical volume using a priori duplication rates. After the data is written to the logical volume, deduplication is completed at the block layer. The deduplicated data and the metadata generated by deduplication are stored separately on two devices, which are called data devices and metadata devices. When writing data, first access the metadata device based on the fingerprint information of the data block, then compare the fingerprint information with the fingerprint information of the stored data, and then deduplicate the duplicate data or store the non-duplicate data in the data device based on the comparison results, and modify the metadata to save the relevant information. To read data, you must first access the metadata device to obtain the address information, and then read it from the data device. The writing process mainly includes the following five steps:

[0068] 1.DMdedup creates a logical volume in the block layer using physical device 1 and physical device 2.

[0069] 2. The file system layer perceives the logical volume and formats the file system on the logical volume. When receiving a write data request from the upper layer, the file system directly passes the data request downward without performing deduplication.

[0070] 3. After the logical volume in the block layer obtains the data block "2A" to be written, its fingerprint information is compared with the fingerprint information of the written data, and whether the data block "2A" is duplicate data is determined based on the comparison result.

[0071] 4. If data block "2A" is duplicate data, the block layer maps data block "2A" to the actual address of data block "2A" written in the data device, and updates the metadata in the metadata device. If data block "2A" is not duplicate data, the block layer writes data block "2A" to the data device normally and adds relevant metadata in the metadata device.

[0072] 5. The block layer writes the only copy to physical device 1 based on the data type, while the metadata generated by deduplication is stored separately in physical device 2.

[0073] However, Dmdedup only supports online deduplication, which makes it unfriendly to some scenarios with high write pressure, such as fsync and direct I / O. Among them, the fsync function is used to synchronize all modified file data in memory to the storage device. In addition, due to deduplication, metadata is stored on a separate device in the external storage. In the process of reading and writing data, it is necessary to first access the metadata device on the critical path to obtain relevant address information before operating on the data device, resulting in serious performance degradation. At the same time, since the logical volume is generated using the prior repetition rate, its size is determined, and Dmdedup's initialization of the logical volume limits the total amount of logical data that can be stored to the maximum extent. In addition, the file system cannot perceive the semantics of the file system layer, which will cause the "false storage space" problem. That is, if the prior repetition rate used is higher than the repetition rate of the actual load, the file system will think that there is still space to store data, but the storage space in the actual device is full, which will cause the upper-layer application and user to fail to write new data. Conversely, when the logical volume is full and the storage space in the actual device is free, it will lead to a waste of storage capacity, and the problem of not being able to fully utilize the actual storage space in the device will occur. In addition, since Dmdedup cannot make good use of the semantics of the file system layer, it will perform unnecessary fingerprint calculations and management overhead on non-data content, such as file mappings and directory indexes with extremely low duplication rates and frequent updates. At the same time, some important metadata will face the risk of data loss, such as duplicate data in the upper-level file system metadata image or update log. The original intention is to use redundancy to avoid sudden power outages or system crashes, but if Dmdedup forcibly deduplicates these important information, it will lead to reduced system security. In addition, Dmdedup chooses to wait for background or offline operations such as garbage collection before removing expired key values. For example, when the system is idle or resource usage is low, some background garbage collection or defragmentation operations may be triggered, and fingerprint calculations and removal of expired key values ​​will also be performed at this time. However, when space consumption is too fast and garbage collection is triggered frequently, the system space recovery performance will be affected.

[0074] SmartDedup is a typical representative of user space deduplication. SmartDedup focuses on lightweight online / offline deduplication combined technical solutions in resource-constrained mobile devices. Deduplication is performed in user space (including user space processes, applications, user space file systems, etc.), redundant writes are eliminated before the data reaches the kernel layer, and address indexes are managed through additional metadata areas, which is transparent to the underlying file system and storage devices. In terms of mapping relationships, SmartDedup allows the same physical address to be stored in the parent node of the file data block, and deduplication is completed by directly mapping the logical address in the file system to the physical address. Its writing process mainly includes the following five steps:

[0075] 1. The file system layer obtains the data block to be written and calculates the fingerprint information of the data block.

[0076] 2. The file system layer performs comparison based on fingerprint information.

[0077] 3. If the comparison result indicates that the data block is duplicate data, the file system layer directly saves the physical address of the copy in the parent node of the duplicate data, for example, storing the physical address pointing to the data block "2A" in node 1 in node 2, and updating the reference count of the data block in the metadata. If the comparison result indicates that the data block is not duplicate data, the file system layer allocates a new physical address and physical block to the data block, and saves the fingerprint information of the data block for subsequent block deduplication. The parent node is used to store the mapping relationship between data and the physical address of the data.

[0078] 4. Write the non-duplicate data normally, such as writing the data block to the physical block, and setting the address of the physical block to the assigned physical address. Since the duplicate data already has a copy in the storage device, the file system layer terminates the actual writing process of the duplicate data.

[0079] 5. Complete writing.

[0080] However, because SmartDedup does not integrate deduplication in the kernel file system, in addition to disk key management, SmartDedup also needs to maintain address indexes on the disk, as well as be responsible for specialized consistency and fragmentation management. It will immediately remove expired key values ​​after the data is invalid, resulting in additional time, computing, and storage overhead. In addition, SmartDedup records new data that has not been deduplicated, which requires a high storage overhead to record the metadata of the new data. When the pre-allocated memory space is exhausted, the system will forcibly trigger offline deduplication, affecting system performance.

[0081] Based on this, an embodiment of the present application provides a data management method. The method is applied to the file system layer. In the data management method, the file system layer obtains a first data block in a first file to be written to an external memory, writes the first data block to the external memory, and adds a first non-deduplicated mark to the first file, and then, based on the first non-deduplicated mark, performs an offline deduplication operation on the first data block.

[0082] In this data management method, the file system layer adds a non-deduplicated mark to the file, and performs offline deduplication operations on the data block based on the non-deduplicated mark, which can achieve incremental deduplication of data, reduce the time and computing overhead of data deduplication, and can also reduce the space overhead caused by incremental data recording in traditional solutions, saving storage space to a large extent. At the same time, since the file system is a user-visible granularity, by adding a non-deduplicated mark to the file, it provides conditions for the user to specify the scope of offline deduplication, which helps to provide users with interactive functions such as file semantic perception and deduplication range definition, so as to meet the personalized needs of users.

[0083] The following first describes an implementation environment involved in a data management method provided in an embodiment of the present application.

[0084] Figure 2 Schematic diagram of an implementation environment involved in a data management method provided in an embodiment of the present application. Figure 2 As shown, the implementation environment includes: a computing device 10. The computing device 10 can obtain a first data block of a first file to be written into an external memory, write the first data block into the external memory, add a first non-deduplicated mark to the first file, and perform an offline deduplication operation on the first data block based on the first non-deduplicated mark. Figure 2 In the illustrated implementation environment, the file to be written may be a file generated by the computing device 10 itself, or a file received by the computing device 10 from the outside.

[0085] In one implementation, the data management method provided in the embodiment of the present application can be implemented by running an executable program on the computing device 10. For example, the executable program of the data management method can be presented in the form of an application installation package. After the application installation package is installed in the computing device 10, the data management method can be implemented by running the executable program. In this case, the computing device 10 can be a terminal. The terminal can be a computer, a personal computer, a portable mobile terminal, a multimedia player, an e-book reader, or a wearable device, etc.

[0086] Figure 3 FIG. 1 is a schematic diagram of an implementation environment involved in another data management method provided in an embodiment of the present application. Figure 3 As shown, the implementation environment may include: a client 20 and a computing device 10. The computing device 10 is used to store data and execute the data management method provided in the embodiment of the present application. The client 20 can establish a communication connection with the computing device 10. For example, a communication connection can be established between the client 20 and the computing device 10 through a network. Optionally, the network can be a local area network, the Internet, or other networks, which is not limited in the embodiment of the present application.

[0087] The client 20 is used for the user to interact with the computing device 10. In one implementation, the client 20 is used to send instructions to the computing device 10 according to the user's instructions. For example, the client 20 is used to send a data write instruction to the computing device 10 according to the user's instructions to instruct the writing of data to the external memory. For another example, the client 20 is used to send a data read instruction to the computing device 10 according to the user's instructions to instruct the reading of data from the external memory. The computing device 10 is used to receive the instruction sent by the client 20 and perform the operation indicated by the instruction. For example, the computing device 10 is used to receive a data write instruction, and according to the data management method provided in the embodiment of the present application, the first data block is stored in the external memory based on the data write instruction. For another example, the computing device 10 is used to receive a data read instruction, and according to the data management method provided in the embodiment of the present application, the target data is obtained based on the data read instruction, and the target data is fed back to the client 20.

[0088] In one implementable manner, the client 20 may be a desktop computer, a laptop computer, a mobile phone, a smart phone, a tablet computer, a multimedia player, a smart home appliance, an artificial intelligence device, a smart wearable device, an e-reader, a smart vehicle-mounted device, or an Internet of Things device. The computing device 10 may be a server (such as a cloud server). Typically, the computing device 10 may be a server cluster consisting of several servers, or may be implemented by a cloud computing service center. Among them, a large number of basic resources owned by a cloud service provider are deployed in the cloud computing service center. For example, computing resources, storage resources, and network resources are deployed in the cloud computing service center. The cloud computing service center can utilize this large amount of basic resources to implement the data management method provided in the embodiment of the present application.

[0089] When the computing device 10 is implemented through a cloud computing service center. The user can access the cloud platform through the client 20, and use the storage service provided by the external memory through the cloud platform. At this time, the functions implemented by the data management method provided in the embodiment of the present application can be abstracted by the cloud service provider into a storage cloud service on the cloud platform, and the cloud platform can use the resources in the cloud computing center to provide the storage cloud service to the user. After the user purchases the storage cloud service on the cloud platform, the storage cloud service can store the data that needs to be written for the user, or provide the stored data to the user. Optionally, the cloud platform can be a cloud platform of a central cloud, a cloud platform of an edge cloud, or a cloud platform including a central cloud and an edge cloud, and the embodiment of the present application does not specifically limit it. It should be noted that in Figure 3 In the implementation environment shown, the computing device 10 can also be implemented by other resource platforms besides the cloud platform, which is not specifically limited in the present embodiment. In this case, the computing device 10 can be implemented by resources in other resource platforms and provide relevant storage services to users.

[0090] It should be understood that the above content is an exemplary description of the application scenario of the data management method provided in the embodiment of the present application, and does not constitute a limitation on the application scenario of the data management method. A person of ordinary skill in the art will know that as business needs change, its application scenario can be adjusted according to application needs. For example, the data management method provided in the embodiment of the present application can be applied to a computing device deployed with F2FS, but is not limited to being applied to a computing device deployed with F2FS, and can also be applied to other file systems with the same technical problems. In addition, when the data management method provided in the embodiment of the present application is applied to a computing device deployed with F2FS, it inherits the function of deduplication into F2FS, and the file system can be regarded as a flash-friendly deduplication file system (F2DFS) modified on the basis of F2FS.

[0091] The data management method provided in the embodiment of the present application is applied to Figure 3 Taking the application scenario shown as an example, the implementation process of the data management method provided by the embodiment of the present application is described. The data management method mainly involves the process of deleting duplicate data and deleting data based on user instructions. The deleting duplicate data process is described below.

[0092] Figure 4 1 is a flow chart of a data deduplication process provided by an embodiment of the present application. Figure 4 As shown, the data writing process includes the following steps:

[0093] Step 401: The file system layer obtains a first data block in a first file to be written into an external storage device.

[0094] The file to be written may be a file generated by the computing device itself, or a file received by the computing device from outside. For example, the file to be written may be a file indicated by a write request sent by a user. In one implementation, the file system layer may process the data in the file in units of data blocks, and the data blocks may be obtained by dividing the data in the file. The embodiment of the present application is described by taking the file system layer processing the data blocks of a file as an example.

[0095] Step 402: The file system layer determines whether the first data block is suitable for performing an online deduplication operation.

[0096] The file system layer writes the file to the external storage, which is the process of writing the multiple data blocks included in the file to the external storage. Before the file system layer writes each data block to the external storage, it needs to first determine whether the data block is suitable for online deduplication operation, so as to perform the corresponding write operation on the data block according to the judgment. For example, during the writing process, the online deduplication operation is performed on the data block that is suitable for online deduplication operation, and the data block that is not suitable for online deduplication operation is first written to the external storage, and then the offline deduplication operation is performed on the data block at an appropriate time.

[0097] In one implementation, before the file system layer writes the first data block to the external memory, it can determine whether the first data block is suitable for performing an online deduplication operation according to the current write mode and / or the load of the file system. For example, when the process of performing the write operation in the write mode is incompatible with the execution process of deduplication, there is no need to perform online deduplication on the first data block. For example, when the write mode writes data by directly flushing data (such as naked input and output (Direct IO)), since the write mode is incompatible with the execution process of deduplication, the first data block is not suitable for performing an online deduplication operation. The read and write process of Direct IO does not pass through the page cache, and directly reads and writes to the external memory, while the execution of online deduplication needs to be performed on the page cache. For another example, when the overhead of performing online deduplication on the first data block in the write mode does not match the benefit, the first data block is not suitable for performing an online deduplication operation. For example, when the current repetition rate of the file system is low, if online deduplication is performed on each data block to be written, the benefit obtained may not be enough to compensate for the consumption of memory and computing resources (such as central processing unit (CPU)) generated by online deduplication, and at this time, the first data block is not suitable for performing an online deduplication operation. For another example, when the write mode indicates that the data block currently being written is sensitive to latency, the first data block is not suitable for performing an online deduplication operation. For another example, when the load of the file system exceeds a specified threshold when preparing to write the first data block, the first data block is not suitable for performing an online deduplication operation.

[0098] Step 403: When the first data block is suitable for performing an online deduplication operation, the file system layer writes the first data block into an external memory, and performs an online deduplication operation on the first data block during the writing process.

[0099] After determining that the first data block is suitable for performing an online deduplication operation, the file system layer can start the writing process and perform an online deduplication operation on the first data block during the writing process. The file system layer first determines whether the first data block is duplicate data, and then performs an online deduplication operation on the first data block according to whether the first data block is duplicate data. When the first data block is not duplicate data, the file system layer allocates a physical address to the first data block, records the first data block in the physical address, records the correspondence between the physical address and the logical address of the first data block, and updates the key value of the first data block to the external memory. Among them, the key in the key value of the first data block can be selected as the fingerprint information of the first data block, and the value can be selected as the physical address of the first data block. When the first data block is duplicate data, the file system layer obtains the physical address of the duplicate data of the first data block, and records the correspondence between the physical address of the first data block and the logical address of the first data block. In addition, in order to manage the consistency of the written data, the file system layer can also record or update the total number of logical addresses corresponding to the physical address each time the data is written. For example, since the total number of logical addresses corresponding to the physical addresses of the duplicate data has been recorded during the writing process of the duplicate data, when the first data block is duplicate data, the total number of logical addresses corresponding to the physical addresses is updated based on the total number of logical addresses corresponding to the physical addresses originally recorded. Here, updating the total number actually means adding one to the original total number. When the first data block is not duplicate data, the total number of logical addresses corresponding to the physical addresses of the first data block is recorded as one. The logical address is used to indicate the position of the first data block in the file.

[0100] In one implementation, the file system layer may obtain fingerprint information of the first data block, and determine whether the first data block is duplicate data based on the fingerprint information of the first data block. Figure 5 A schematic diagram of determining whether a first data block is duplicate data is provided in an embodiment of the present application. Figure 5 As shown, the implementation process includes:

[0101] Step 4031: Obtain fingerprint information of the first data block.

[0102] The fingerprint information of the first data block is a unique identifier of the first data block. The fingerprint information of the first data block can be obtained based on the hash value of the first data block. For example, the fingerprint information of the first data block can be obtained by performing anti-collision hash calculation on the first data block. Anti-collision means that the probability of repeated occurrence of the generated hash value is low. For example, when writing a file to an external storage system, since the data of the file can be cached in the page cache of the memory first, the hash calculation can be performed in units of pages in the page cache, and the result of the hash calculation is the fingerprint information of the first data block. Optionally, when performing the hash calculation in units of pages in the page cache, the data stored in the page can also be divided according to a specified size (such as 4 kilobytes (KB)), and the corresponding hash value is obtained based on the data obtained by the division. Among them, the hash algorithm used in the present application can be adjusted according to the application requirements, for example, it can be a message digest algorithm version 5 (message-digestalgorithm 5, MD5) algorithm, a secure hash algorithm (secure hash algorithm, SHA), etc.

[0103] Step 4032: Match the fingerprint information of the first data block with the fingerprint information of the written data stored in the memory to determine whether the first data block is duplicate data.

[0104] After the fingerprint information of the first data block is obtained, the fingerprint information can be matched with the fingerprint information of the written data. Since the fingerprint information is used to uniquely identify the data, when the fingerprint information of the first data block matches the fingerprint information of the written data, it can be considered that the first data block is the same as the written data, and the first data block is determined to be duplicate data. When the fingerprint information of the first data block does not match the fingerprint information of the written data, it can be considered that the first data block is different from the written data, and the first data block is determined not to be duplicate data. Among them, the fingerprint information of the first data block matches the fingerprint information of the written data, which can be the fingerprint information of the first data block is the same as the fingerprint information of the written data. The fingerprint information of the data can quickly identify the index of the data content, and its efficiency is much higher than the method of comparing the byte-level data content, which can ensure the efficiency of judging whether the first data block is a duplicate data block. In addition, since its collision probability (false positive) is extremely low, even much lower than the hardware error probability, it can ensure the accuracy of judging whether the first data block is a duplicate data block.

[0105] In an embodiment of the present application, the fingerprint information of all written data is stored in the memory. After the fingerprint information of the first data block is obtained, it is possible to search in the memory whether the same fingerprint exists based on the fingerprint information of the first data block. When there is a fingerprint identical to the fingerprint information of the first data block in the memory, it is determined that the fingerprint information of the first data block matches the fingerprint information of the written data. When there is a fingerprint different from the fingerprint information of the first data block in the memory, it is determined that the fingerprint information of the first data block does not match the fingerprint information of the written data. At this time, the fingerprint information of the first data block can also be written into the memory.

[0106] In one implementation, the memory can manage the fingerprint information according to the prefix of the fingerprint information. For example, the memory maintains multiple memory sets (such as memory buckets), and the multiple memory sets correspond to multiple prefixes respectively, and any memory set is used to store all fingerprint information with corresponding prefixes. Among them, the size of the memory set can be scaled according to the device resources. After obtaining the fingerprint information of the first data block, it can be determined whether there is a memory set corresponding to the prefix in the memory based on the prefix of the fingerprint information. When the memory set corresponding to the prefix does not exist in the memory, it is determined that the fingerprint information of the first data block does not match the fingerprint information of the written data. When the memory set corresponding to the prefix exists in the memory, continue to search whether the fingerprint information exists in the memory set. When the fingerprint information does not exist in the memory set, it is determined that the fingerprint information of the first data block does not match the fingerprint information of the written data. When the fingerprint information exists in the memory set, it is determined that the fingerprint information of the first data block does not match the fingerprint information of the written data. For example, Figure 6 As shown, the memory maintains multiple memory buckets, one of which corresponds to the prefix B452, and the memory bucket is used to store all fingerprint information with the prefix B452. After obtaining the fingerprint information B4522F02F35C5340F620D37AD66434D9D3CEB69C of the first data block, it can be obtained that the prefix of the fingerprint information is B452. Then, the memory is accessed with the prefix B452 as the key to search for the pointer of the location of the fingerprint bucket, so as to obtain the memory bucket corresponding to B452 in the memory. Then continue to search whether there is fingerprint information B4522F02F35C5340F620D37AD66434D9D3CEB69C in the memory bucket corresponding to B452. Since the fingerprint information exists in the memory bucket shown in 10, it is determined that the fingerprint information of the first data block is the same as the fingerprint information of the written data, and the first data block is duplicate data. Among them, the prefix of the fingerprint information can be represented by the characters of the specified number of digits from left to right in the fingerprint information. The designated number of bits can be set according to application requirements, for example, the designated number of bits is four.

[0107] It should be noted that the memory can also manage the fingerprint information according to the access heat of the written data. For example, the memory stores the fingerprint information of the written data in order from high to low access heat of the written data. Then, each time the fingerprint information of the first data block matches the fingerprint information of the written data, the order of the fingerprint information of the first data block in the memory can be adjusted accordingly. For example, when the fingerprint information of the first data block matches the fingerprint information of the written data, it can be considered that the fingerprint information has the highest current heat, and the fingerprint information can be moved to the head of the memory bucket. In addition, when the memory bucket is full, the fingerprint information at the end of the memory bucket can be deleted. By managing the fingerprint information according to the access heat, it is possible to quickly find recently written data and remove long-term unused data in the memory when writing data.

[0108] Step 404: When the first data block is not suitable for performing an online deduplication operation, the file system layer writes the first data block into an external memory and adds a first non-deduplicated mark to the first file.

[0109] When the first data block is not suitable for performing online deduplication operations, it means that the first data block needs to be deduplicated through offline deduplication operations. The file system layer first writes the first data block to an external memory, and adds a first non-deduplicated mark to the first file to which the first data block belongs, so as to perform offline deduplication operations on the first data block based on the first non-deduplicated mark.

[0110] In one implementation, the file system layer maintains a queue to be offlined, which is used to record identification information of files that need to be deduplicated offline. That is, all files recorded in the queue to be offline have data blocks that need to be deduplicated offline. Figure 7 It is a schematic diagram of a queue to be offline provided by an embodiment of the present application. The queue to be offline includes a plurality of queue nodes arranged in order, and each queue node is used to record the identification information of the file that needs to be deduplicated offline by itself. The implementation method of adding a first non-deduplicated mark to the first file includes: adding the identification information of the first file to the queue to be offline. Recording the files that need to perform offline deduplication operations in the queue to be offline helps to quickly determine the files that need to perform offline deduplication operations during the subsequent offline deduplication operations, thereby improving the execution speed of offline deduplication.

[0111] It should be noted that the identification information of the file may not be limited to being stored in a queue, but may also be stored in other ways. For example, the file system layer may optionally maintain a pending offline set, which is used to record the identification information of the files that need to perform offline deduplication operations. Alternatively, a first non-deduplicated mark may be added to the first file by tagging the file.

[0112] Optionally, according to the previous description, before the file system layer writes each data block to the external memory, it must determine whether the data block is suitable for online deduplication operations. For multiple data blocks in a file, some of the multiple data blocks may be suitable for online deduplication operations, while some of the data blocks may not be suitable for online deduplication operations. Therefore, when the first data block is not suitable for online deduplication operations, the file system layer can also add a second non-deduplicated mark to the first data block to mark that the first data block needs to perform an offline deduplication operation. The method also includes: the file system layer adds a second non-deduplicated mark to the first data block.

[0113] In one implementation, the file system layer maintains a to-be-offline bitmap for each file. The to-be-offline bitmap includes a plurality of bits, and the plurality of bits correspond to a plurality of data blocks of the file. The value recorded in each bit is used to indicate whether the data block corresponding to the bit needs to perform an offline deduplication operation. For example, Figure 8 As shown, when the first file has 100 data blocks, the file system maintains a bitmap with 100 bits for the first file, and the 100 bits correspond one-to-one to the 100 data blocks. Figure 8 Each square in represents a bit. When the value recorded in the bit is 0, it indicates that the data block corresponding to the bit does not need to perform offline deduplication operation. When the value recorded in the bit is 1, it indicates that the database corresponding to the bit needs to perform offline deduplication operation. Assuming that the initial value recorded in each bit of the offline bitmap of the file is 0, the implementation method of adding a second non-deduplication mark to the first data block includes: determining the bit corresponding to the first data block in the offline bitmap, and changing the value recorded in the bit from 0 to 1. By using the offline bitmap to record the data blocks that need to perform offline deduplication operation, it is helpful to quickly determine the data blocks that need to perform offline deduplication operation in the subsequent offline deduplication operation process, thereby improving the execution speed of offline deduplication. In addition, by maintaining the offline bitmap for each file instead of maintaining a global scope bitmap for all data recorded in the external memory, on the one hand, the search range of the data blocks that need to perform offline deduplication operation is narrowed, and the storage space overhead, time overhead and calculation overhead of searching for the data blocks that need to perform offline deduplication operation are reduced, and on the other hand, the user's interactivity in the process is enhanced.

[0114] It should be noted that adding the second non-deduplicated mark to the first data block may not be limited to the offline bitmap recording method, and may also be recorded in other ways. For example, the file system layer may optionally maintain a to-be-offline set for each file, and the to-be-offline set is used to record the identification information of the data blocks in the file that need to perform offline deduplication operations.

[0115] It should be noted that for data blocks suitable for online deduplication operations, if it is determined during the online deduplication operation that the data block is not duplicate data, a non-deduplicated mark can also be added to the data block and its corresponding file to facilitate offline deduplication operations on the data block during the offline deduplication operation to remove potential duplicate data in the file system.

[0116] Step 405: The file system layer performs an offline deduplication operation on the first data block based on the first non-deduplicated mark.

[0117] After the file system layer writes the first data block to the external storage device, it can perform offline deduplication operation on the first data block at an appropriate time. For example, when the file system is idle, the file system layer performs offline deduplication operation on all data blocks that need to be deduplicated offline. The implementation process includes:

[0118] Step 405a: Determine a first data block based on the first non-deduplicated mark.

[0119] According to the description of step 404, when the first data block is not suitable for online deduplication operation, there are two cases for adding a non-deduplicated mark. The implementation process of step 405a is described below for these two cases respectively.

[0120] In the first case, when the file system layer only adds the first non-deduplication mark to the file, the data block that needs to perform offline deduplication operation can be selected as any data block in the file with the first non-deduplication mark added. That is, when the non-deduplication mark is not added to the data block in the file, offline deduplication operation can be selected for all data blocks in the file. The implementation process of step 405a includes: step a11, based on the first non-deduplication mark, determine the first file from multiple files written to the external memory. Among them, the file with the first non-deduplication mark added is the first file, that is, the file that needs to perform offline deduplication operation. Step a12, determine each data block in the first file as a first data block, and the first data block needs to perform offline deduplication operation.

[0121] In the second case, when the file system layer adds a first non-deduplication mark to the file and adds a second non-deduplication mark to the data block, the data block that needs to perform offline deduplication operation can be selected as the data block with the second non-deduplication mark added in the file with the first non-deduplication mark added. Then the implementation process of step 405a includes: step a21, based on the first non-deduplication mark, determine the first file from multiple files written to the external memory. Among them, the file with the first non-deduplication mark is the first file, that is, the file that needs to perform offline deduplication operation. Step a22, based on the second non-deduplication mark, determine the first data block from multiple data blocks in the first file. Among them, the data block with the second non-deduplication mark added is the first data block, that is, the data block that needs to perform offline deduplication operation.

[0122] The following is an example of an implementation process of determining a data block that needs to perform an offline deduplication operation based on a first non-deduplication mark and a second non-deduplication mark. Fig. 9 As shown, when the file system layer maintains a queue to be offline and a bitmap to be offline, during the execution of step 405a, the file system layer may first execute Fig. 9 ① in the above method is to read the queue to be offline into the memory and identify each file recorded in the queue to be offline as a file that needs to perform offline deduplication operation. Then, the file system layer executes Fig. 9 In ②, based on the ino and pid information stored in the offline queue, determine the file that needs to perform offline deduplication operation, and obtain the offline bitmap of the file based on the file mapping. Among them, the ino and pid information are the identification information used to indicate the file. Then, the file system layer executes Fig. 9 ③, based on the offline bitmap to be performed on the file, the identification information of the data block that needs to perform offline deduplication operation is obtained, and according to the identification information, the data block that needs to perform offline deduplication operation is read from the external storage device into the memory, thereby obtaining the data block that needs to perform offline deduplication operation.

[0123] Step 405b: perform an offline deduplication operation on the first data block.

[0124] When performing an offline deduplication operation on the first data block, the file system layer first determines whether the first data block is duplicate data, and then performs corresponding processing operations on the first data block based on whether the first data block is duplicate data. In one implementation, the file system layer can obtain fingerprint information of the first data block, and determine whether the first data block is duplicate data based on the fingerprint information of the first data block. Please refer to the relevant description in the corresponding operation step 4031 for the method of obtaining the fingerprint information of the first data block, which will not be repeated here. Fig.10 A schematic diagram of determining whether a first data block is duplicate data is provided in an embodiment of the present application. Fig.10 As shown, the implementation process includes:

[0125] Step b1: Match the fingerprint information of the first data block with the fingerprint information stored in at least one of the internal memory and the external memory.

[0126] The external memory contains fingerprint information of all data written into the external memory. Matching the fingerprint information of the first data block with the fingerprint information stored in the external memory means matching the fingerprint information of the first data block with the fingerprint information of the full amount of data stored in the external memory, which can ensure the accuracy of determining whether the first data block is duplicate data. The internal memory contains fingerprint information of data with high access popularity. Matching the fingerprint information of the first data block with the fingerprint information stored in the internal memory means matching the fingerprint information of the first data block with the fingerprint information of data with high access popularity. Since the internal memory has a faster access speed and the probability that the fingerprint information stored in the internal memory is the fingerprint information of duplicate data is high, the speed of determining whether the first data block is duplicate data can be guaranteed. Therefore, when determining whether the first data block is duplicate data, it is optional to match the fingerprint information of the first data block with the fingerprint information stored in at least one of the internal memory and the external memory.

[0127] In one implementation, the file system layer matches the fingerprint information of the first data block with the fingerprint information stored in the memory and the external memory, and determines whether the first data block is duplicate data according to the matching result. The implementation process includes:

[0128] Step b11: Match the fingerprint information of the first data block with the fingerprint information stored in the memory.

[0129] According to the previous description, since the memory has a faster access speed and the fingerprint information stored in the memory is more likely to be the fingerprint information of duplicate data, the file system layer may optionally prioritize matching the fingerprint information of the first data block with the fingerprint information already stored in the memory. When the fingerprint information of the first data block matches the fingerprint information already stored in the memory, it is determined that the first data block is duplicate data. At this point, the file system completes the matching process without having to match the fingerprint information of the full amount of data stored in the external memory, which shortens the time consumption of the matching process and can ensure the speed of determining whether the first data block is duplicate data. For example, Fig. 9 As shown in the figure, after the file system layer obtains the data blocks that need to be deduplicated offline, it first executes Fig. 9 ④ in the above, obtain the fingerprint information of the database, and match the fingerprint information with the fingerprint information stored in the memory bucket.

[0130] Step b12: When the fingerprint information stored in the memory does not match the fingerprint information of the first data block, continue to match the fingerprint information of the first data block with the fingerprint information stored in the external memory.

[0131] When the fingerprint information stored in the memory is different from the fingerprint information of the first data block, it means that the first data block does not match the data identified by the fingerprint information stored in the memory. The file system layer then continues to match the fingerprint information of the first data block with the fingerprint information stored in the external storage to match the first data block with the full amount of data stored in the external storage.

[0132] In one implementation, the external memory includes a first-level storage unit and a second-level storage unit. The entries stored in the first-level storage unit are entries removed from the memory. The entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging. In an embodiment of the present application, the external memory maintains the key value of the data through the first-level storage unit and the second-level storage unit. The aforementioned "entry" is the key value corresponding to the data block. The memory, the first-level storage unit and the second-level storage unit constitute a three-layer storage structure of the key value in the present application. For example, Fig.11 A schematic diagram of a three-layer storage structure of a key value provided in an embodiment of the present application, and memory using a memory bucket to manage key values. Fig.11 As shown in the figure, the process of writing key values ​​to external storage includes the following steps:

[0133] ①. When writing new data to the external storage, the file system layer writes the key value to the memory bucket. Optionally, the memory manages the fingerprint information according to the access popularity of the written data. Please refer to the relevant description in the previous content for its management method.

[0134] ②. After the memory bucket is full, the file system layer moves some key values ​​from the memory bucket to the first-level storage unit to persist the part of the key values ​​in the first-level storage unit. When the memory manages the fingerprint information according to the access popularity of the data, the part of the key values ​​removed in this step is the key value located at the tail of the memory bucket. In one implementation, the first-level storage unit manages the key value in a log manner. For example, the first-level storage unit includes a plurality of log nodes arranged in order. When the file system layer writes the key value from the memory to the first-level storage unit, the key value is written to the head of the log node at the head of the team. At this time, the first-level storage unit is also called a key value log. For example, the first-level storage unit arranges a plurality of log nodes according to the write time. When the file system layer writes the key value from the memory to the first-level storage unit, the key value is written to the head of the log node at the head of the team in an append manner.

[0135] ③. At a specified time, the file system layer compresses the key values ​​in the first-level storage unit, and deletes expired key values ​​during the compression process to maintain the consistency of data and key values. Among them, the expired key value is the key value of the deleted data. Optionally, the specified time can be a time when the file system is idle. When the file system layer writes the key value from the memory to the first-level storage unit, the key value is written to the head of the log node at the head of the team. Compressing the key value in the first-level storage unit means compressing the key value stored in the log node at the head of the team, and writing the compressed key value to the log node after the current log node.

[0136] ④. At a specified time, the key value in the first-level storage unit is submitted to the second-level storage unit for merging to stabilize the key value. During the merging process, the file system layer also needs to delete expired key values ​​to maintain the consistency of data and key values. Optionally, the specified time may be when the file system is idle. In one implementation, the second-level storage unit manages key values ​​in a mirrored manner. For example, the second-level storage unit includes a plurality of mirror nodes arranged in order, and when the file system layer writes a key value from the first-level storage unit to the second-level storage unit, the key value is written to the head of the mirror node at the head of the queue. At this time, the second-level storage unit is also referred to as a key value mirror. In an embodiment of the present application, the external memory may maintain a key value log and a key value mirror for each memory bucket.

[0137] When the external memory includes a first-level storage unit and a second-level storage unit, since the first-level storage unit and the second-level storage unit may both record key values, when matching the fingerprint information of the first data block with the fingerprint information stored in the external memory, the fingerprint information of the first data block may be optionally matched with the fingerprint information recorded in the first-level storage unit and the second-level storage unit. In another implementation, the offline deduplication operation is generally performed at a specified time, such as when the file system is idle, and before the specified time, the file system may optionally submit all key values ​​recorded in the first-level storage unit to the second-level storage unit. At this time, since the key value is not recorded in the first-level storage unit when performing the offline deduplication operation, when matching the fingerprint information of the first data block with the fingerprint information stored in the external memory, the fingerprint information of the first data block is actually matched with the fingerprint information recorded in the second-level storage unit. For example, Fig. 9 As shown, when the fingerprint information stored in the memory does not match the fingerprint information of the first data block, the file system layer executes Fig. 9 In step ⑤, continue to match the fingerprint information of the first data block with the fingerprint information stored in the key-value image.

[0138] Step b2: When the fingerprint information of the first data block matches the stored fingerprint information, deduplication processing is performed on the first data block.

[0139] When the fingerprint information of the first data block matches the stored fingerprint information, it indicates that the first data block is duplicate data, and deduplication processing is performed on the first data block. At this time, the file system layer obtains the physical address of the duplicate data of the first data block, and modifies the correspondence between the physical address of the first data block and the logical address of the first data block to correspondence between the logical address of the first data block and the physical address of the duplicate data of the first data block. In addition, the file system also needs to obtain the physical address of the first data block and release the physical data block indicated by the physical address to achieve the purpose of deleting duplicate data. In addition, in order to manage the consistency of the written data, the file system layer can also optionally update the total number of logical addresses corresponding to the physical address. For example, since the total number of logical addresses corresponding to the physical address of the duplicate data has been recorded in the process of writing duplicate data, when the first data block is duplicate data, the total number of logical addresses corresponding to the physical address is updated on the basis of the total number of logical addresses corresponding to the physical address originally recorded. Here, updating the total number is actually adding one to the original total number.

[0140] Step b3: When the fingerprint information of the first data block does not match the stored fingerprint information, the key value of the first data block is updated to the external memory based on the fingerprint information of the first data block.

[0141] When the fingerprint information of the first data block does not match the stored fingerprint information, it means that the first data block is not duplicate data, and the file system layer needs to update the file mapping of the first data block. In addition, when the first data block is not suitable for performing online deduplication operations, since the fingerprint information of the first data block is generated during the offline deduplication process, when the first data block is not duplicate data, the file system layer also needs to update the key value of the first data block to the external memory. Among them, the key in the key value of the first data block can be selected as the fingerprint information of the first data block, and the value can be selected as the physical address of the first data block. Similarly, in order to manage the consistency of the written data, the file system layer can also optionally record the total number of logical addresses corresponding to the physical address. For example, when the first data block is not duplicate data, the total number of logical addresses corresponding to the physical address of the first data block is recorded as one.

[0142] Optionally, the file system may also provide an interactive interface for the user, through which the user may send instructions to instruct the file system to perform relevant operations on the file according to the instructions. Fig.12 As shown, the method may also include:

[0143] Step 406: The file system layer receives a deduplication instruction sent by the user, where the deduplication instruction instructs to perform a deduplication operation on a specified range of the first file.

[0144] When the user needs to specify the scope of the deduplication operation on the first file, a deduplication instruction can be sent to the file system layer to indicate the scope. For example, when the user instructs the first file to be stored on the computing device through the client used by the user, when the user needs to specify the range, the user can send a deduplication instruction to the computing device through the client used by the user, so that the file system layer of the computing device obtains information for performing a deduplication operation on the specified range of the first file through the deduplication instruction. Optionally, the specified range is the first file, the file set to which the first file belongs, the directory of the first file, or the file cluster of the first file. Among them, the file set can be divided according to application requirements. For example, files belonging to the same category can be divided into the same file set according to the type of the file. It should be noted that the user indicating the scope of the deduplication operation through the deduplication instruction is an application method of the interactive interface. Those skilled in the art can know that the interactive interface can have other applications, which are not listed here one by one.

[0145] When the user performs a deduplication range, such as Fig.12 As shown, the implementation process of step 405 includes:

[0146] Step 4051: Based on the deduplication instruction, determine a specified range in multiple files marked with a first non-deduplication mark, and perform an offline deduplication operation on the data blocks included in the specified range.

[0147] According to the previous description, when the file system layer only adds the first non-deduplication mark to the file, the data block that needs to perform offline deduplication operation can be selected as any data block in the file with the first non-deduplication mark added. When determining the data block that needs to perform offline deduplication operation based on the deduplication instruction, if the data range indicated by the first non-deduplication mark is larger than the specified range indicated by the deduplication instruction, the data block indicated by the intersection of the data range indicated by the first non-deduplication mark and the specified range indicated by the deduplication instruction is determined as the data block that needs to perform offline deduplication operation. If the data range indicated by the first non-deduplication mark is smaller than the specified range indicated by the deduplication instruction, the data block indicated by the specified range is determined as the data block that needs to perform offline deduplication operation.

[0148] Similarly, when the file system layer adds a first non-deduplication mark to a file and adds a second non-deduplication mark to a data block, the data block that needs to perform offline deduplication operation can be selected as the data block with the second non-deduplication mark added in the file with the first non-deduplication mark added. When determining the data block that needs to perform offline deduplication operation based on the deduplication instruction, if the data range indicated by the first non-deduplication mark and the second non-deduplication mark is greater than the specified range indicated by the deduplication instruction, the data block indicated by the intersection of the data range indicated by the first non-deduplication mark and the second non-deduplication mark and the specified range indicated by the deduplication instruction is determined as the data block that needs to perform offline deduplication operation. If the data range indicated by the first non-deduplication mark and the second non-deduplication mark is less than the specified range indicated by the deduplication instruction, the data block indicated by the specified range is determined as the data block that needs to perform offline deduplication operation. For example, assuming that the data range indicated by the first non-deduplication mark and the second non-deduplication mark is all data blocks in file a, and the specified range indicated by the deduplication instruction is file a in directory 1, the data blocks in directory 1 of file a are determined as data blocks that need to perform offline deduplication operation.

[0149] As can be seen from the above, in this data management method, the file system layer can achieve incremental deduplication of data by adding a non-deduplicated mark to the file and performing offline deduplication operations on the data block based on the non-deduplicated mark, thereby reducing the time and computing overhead of data deduplication, and can also reduce the space overhead caused by incremental data recording in traditional solutions, saving storage space to a large extent. At the same time, since the file system is a user-visible granularity, by adding a non-deduplicated mark to the file, it provides conditions for the user to specify the scope of offline deduplication, which helps to provide users with interactive functions such as file semantic perception and deduplication range definition, so as to meet the personalized needs of users.

[0150] In addition, when a data block is not suitable for online deduplication operations, the present application combines the advantages of online deduplication and offline deduplication by performing offline deduplication operations on the data block, and avoids their respective limitations, thereby maximizing the benefits of duplicate data deletion.

[0151] The above describes the deduplication of the file system. The following describes the process of deleting data based on user instructions. Fig.13 1 is a flowchart of a process of deleting data based on user instructions provided by an embodiment of the present application. Fig.13 As shown, the process includes the following steps:

[0152] Step 1301: The file system layer receives a deletion instruction instructing to delete a second data block.

[0153] When the user needs to delete the second data block in the first file, a deletion instruction may be sent to the file system layer to indicate that the second data block needs to be deleted. For example, the user sends a deletion instruction to the computing device through the client used by the user, so that the file system layer of the computing device determines that the second data block needs to be deleted through the deletion instruction. In one implementation, the deletion instruction may indicate that the second data block needs to be deleted through a logical address.

[0154] Step 1302: The file system layer deletes the second data block based on the deletion instruction.

[0155] After the file system layer determines that the second data block needs to be deleted, it can obtain the physical address of the second data block, and then reduce the total number of logical addresses corresponding to the physical address of the second data block by one. When the second data block is duplicate data, the purpose of deleting the second data block is achieved. When the second data block is not duplicate data, it is also necessary to release the physical space indicated by the physical address to achieve the purpose of deleting the second data block. In one implementation, when the deletion instruction indicates that the second data block needs to be deleted through the logical address, the file system layer can query the correspondence between the logical address and the physical address based on the logical address of the second data block to obtain the physical address of the second data block.

[0156] Step 1303: The file system layer deletes the key value of the second data block.

[0157] The process of deleting the key value of the second data block is the process of deleting the expired key value. Fig.14 As shown, the implementation process includes:

[0158] Step 13031: The file system layer obtains the storage location of the key value of the second data block.

[0159] After the file system layer determines that the second data block needs to be deleted, it can obtain the storage location of the key value of the second data block, and perform the corresponding deletion operation according to the actual situation of its storage location. For example, when the deletion instruction indicates that the second data block needs to be deleted through the logical address, the file system layer can query the correspondence between the logical address and the physical address according to the logical address of the second data block, obtain the physical address of the second data block, and query the storage location of the key value with the physical address as the value according to the physical address. In one implementation, it is possible to query whether the key value with the physical address as the value is recorded in the internal memory and the external memory respectively according to the physical address, and when the key value is recorded in the internal memory, it is determined that the key value of the second data block is stored in the internal memory, and when the key value is recorded in the external memory, it is determined that the key value of the second data block is stored in the external memory.

[0160] Step 13032: When the key value of the second data block is stored in the memory and not stored in the external memory, the file system layer deletes the key value of the second data block in the memory.

[0161] When the key value of the second data block is stored in the memory and not in the external memory, it means that the key value of the second data block has not been persisted in the external memory. After the file system layer deletes the key value of the second data block in the memory, the purpose of deleting the key value of the second data block is achieved.

[0162] Step 13033: When the key value of the second data block is not stored in the memory but stored in the external memory, a deletion event indicating deletion of the key value of the second data block is recorded, and the file system layer executes the deletion event at a specified time to delete the key value of the second data block in the external memory.

[0163] When the key value of the second data block is not stored in the memory but is stored in the external memory, since the key value occupies memory resources, the deletion event can be recorded first, and then the deletion event is executed at a specified time, without immediately deleting the key value of the second data block in the external memory. Optionally, the specified time can be a time when the file system is idle. In one implementation, when the key value of the second data block is not stored in the memory but is stored in the external memory, the file system layer can first generate a deletion log and temporarily store the deletion log in the memory, and the deletion log is used to record the key value of the second data block that needs to be deleted. According to the previous description of the memory management method, when the memory bucket is full, the data with low access heat in the memory will be moved out to the first-level storage unit. Therefore, the deletion log will be subsequently moved out to the first-level storage unit. When the first-level storage unit processes the data recorded therein, if the key value of the second data block is stored in the first-level storage unit, the first-level storage unit will delete the key value of the second data block according to the deletion log. If the key value of the second data block is not stored in the first-level storage unit, the first-level storage unit will subsequently move the deletion log out to the second-level storage unit. Similarly, when the second-level storage unit merges the data submitted by the first-level storage unit, it will delete the key value of the second data block according to the deletion log, so as to achieve the purpose of deleting the key value of the second data block in the external storage device.

[0164] Step 13034: When the key value of the second data block is stored in the memory and in the external storage device, the file system layer deletes the key value of the second data block in the memory and the external storage device.

[0165] When the key value of the second data block is stored in the memory and in the external memory, the file system layer needs to store the key value of the second data block in the memory and delete the key value of the second data block in the external memory. In one implementation, after the file system layer determines that the key value of the second data block is stored in the memory and the external memory, it can immediately delete the key value of the second data block from the memory and record a deletion event indicating the deletion of the key value of the second data block. The file system layer executes the deletion event at a specified time to delete the key value of the second data block in the external memory. For the process of deleting the key value of the second data block in the external memory based on the deletion event, please refer to the relevant description in step 13033, which will not be repeated here.

[0166] As can be seen from the above, when data needs to be deleted, the file system layer can immediately release the data reference and reclaim the storage space. When the expired key value is stored in the external storage, the file system layer does not need to delete the expired key value from the external storage immediately, but records the deletion event indicating the deletion of the expired key value, and deletes the expired key value based on the deletion event at a specified time, releasing the space occupied by the expired key value, thus avoiding the high overhead caused by the need to immediately delete the expired key value.

[0167] It should be noted that the sequence of steps of the data management method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0168] The above describes the data management method of the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a data management device. The device is applied to the file system layer. The structural diagram of the data management device is shown in FIG. Fig.15 It should be understood that the system may include more additional nodes than the structure shown or omit some of the nodes shown therein, and the present application embodiment does not limit this. Fig.15 As shown, the data management device 150 includes:

[0169] The acquisition module 1501 is used to acquire a first data block in a first file to be written into an external memory.

[0170] The writing module 1502 is used to write the first data block into the external memory and add a first non-deduplication mark to the first file.

[0171] The deduplication module 1503 is configured to perform an offline deduplication operation on the first data block based on the first non-deduplicated mark.

[0172] Optionally, the writing module 1502 is further configured to add a second non-deduplicated mark to the first data block.

[0173] Optionally, the deduplication module 1503 is specifically used to: determine a first file from multiple files written to an external memory based on a first non-deduplication mark; determine a first data block from multiple data blocks in the first file based on a second non-deduplication mark; and perform offline deduplication operations on the first data block.

[0174] Optional, such as Fig.16 As shown, the data management device 150 also includes:

[0175] The interaction module 1504 is configured to receive a deduplication instruction sent by a user, where the deduplication instruction indicates to perform a deduplication operation on a specified range of the first file;

[0176] The deduplication module 1503 is specifically configured to determine a specified range in a plurality of files marked with a first non-deduplication mark based on a deduplication instruction, and perform an offline deduplication operation on data blocks included in the specified range.

[0177] Optionally, the specified range is the first file, a file set to which the first file belongs, a directory of the first file, or a file cluster of the first file.

[0178] Optionally, the deduplication module 1503 is specifically used to: determine the first data block based on the first non-deduplicated mark; match the fingerprint information of the first data block with the fingerprint information stored in at least one of the internal memory and the external memory; when the fingerprint information of the first data block matches the stored fingerprint information, perform deduplication processing on the first data block; when the fingerprint information of the first data block does not match the stored fingerprint information, update the key value of the first data block to the external memory based on the fingerprint information of the first data block.

[0179] Optionally, the deduplication module 1503 is specifically used to: match the fingerprint information of the first data block with the fingerprint information stored in the memory; when the fingerprint information stored in the memory does not match the fingerprint information of the first data block, continue to match the fingerprint information of the first data block with the fingerprint information stored in the external memory.

[0180] Optionally, the external memory includes a first-level storage unit and a second-level storage unit; the entries stored in the first-level storage unit are entries removed from the memory; the entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging.

[0181] Optional, such as Fig.16 As shown, the data management device 150 also includes:

[0182] The interaction module 1504 is configured to receive a deletion instruction instructing to delete the second data block.

[0183] The acquisition module 1501 is used to acquire the storage location of the key value of the second data block.

[0184] The deleting module 1505 is configured to delete the key value of the second data block in the memory when the key value of the second data block is stored in the memory and is not stored in the external memory.

[0185] The deletion module 1505 is also used to record a deletion event indicating deletion of the key value of the second data block when the key value of the second data block is not stored in the memory but is stored in the external memory, and execute the deletion event at a specified time to delete the key value of the second data block in the external memory.

[0186] The deleting module 1505 is further configured to delete the key value of the second data block in the memory and the external memory when the key value of the second data block is stored in the memory and in the external memory.

[0187] Optional, such as Fig.16 As shown, the data management device 150 further includes: a judgment module 1506, which is used to judge whether the first data block is suitable for performing an online deduplication operation.

[0188] The writing module 1502 is specifically used to write the first data block into the external memory and add a first non-deduplicated mark to the first file when the first data block is not suitable for performing the online deduplication operation.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each node described above can refer to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0190] The embodiment of the present application provides a computing device. The computing device is used to implement some or all of the functions in the data management method provided in the embodiment of the present application. For example, it executes the functions of a management node or a computing node in the data management method. Fig.17 Schematic diagram of a computing device provided in an embodiment of the present application. Fig.17 As shown, the computing device 1700 includes a processor 1701, a memory 1702, a communication interface 1703 and a bus 1704. The processor 1701, the memory 1702 and the communication interface 1703 are connected to each other through the bus 1704.

[0191] Processor 1701 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. A dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a network processor (NP). Processor 1701 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the data management method of the present application may be completed by the hardware integrated logic circuit in the processor 1701 or instructions in the form of software.

[0192] The memory 1702 is used to store computer programs, and the computer programs include an operating system 1702a and executable codes (i.e., program instructions) 1702b. The memory 1702 is, for example, a read-only memory or other types of static storage devices that can store static information and instructions, or a random access memory or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory 1702 is used to store an outbound port queue, etc. The memory 1702 is, for example, independent and connected to the processor 1701 via a bus 1704. Or the memory 1702 and the processor 1701 are integrated together. The memory 1702 may store executable code. When the executable code stored in the memory 1702 is executed by the processor 1701, the processor 1701 is used to perform some or all functions of the data management method provided in the embodiment of the present application. For the implementation of the process executed by the processor 1701, please refer to the relevant description in the aforementioned embodiment. The memory 1702 may also include software modules and data required for other running processes such as an operating system.

[0193] The communication interface 1703 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 1703 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0194] The bus 1704 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the memory 1702, the processor 1701, and the communication interface 1703). For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device through the bus 1704 as an example. Optionally, the above-mentioned devices inside the computing device 1700 can also be connected to each other in communication with each other using other connection methods other than the bus 1704. For example, the above-mentioned devices inside the computing device 1700 are interconnected through an internal logical interface.

[0195] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0196] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing the program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the data management method provided in the embodiments of the present application are implemented.

[0197] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0198] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0199] Optionally, the structure of at least one computing device included in the computing device cluster can be found in Fig.17 The computing device 1700 is shown. The memory 1702 in one or more computing devices 1700 in the computing device cluster may store the same instructions for executing the data management method.

[0200] In some possible implementations, the memory 1702 of one or more computing devices 1700 in the computing device cluster may also store some instructions for executing the data management method. In other words, the combination of one or more computing devices 1700 may jointly execute the instructions for executing the data management method.

[0201] It should be noted that the memory 1702 in different computing devices 1700 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data management device. That is, the instructions stored in the memory 1702 in different computing devices 1700 can implement the functions of one or more modules in the above modules.

[0202] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.18 A possible implementation is shown. Fig.18 As shown, two computing devices 1800A and 1800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, computing devices 1800A and 1800B include a bus 1802, a processor 1804, a memory 1806, and a communication interface 1808. For example, computing device 1800A can perform the function of a management node in the embodiment of the present application, and computing device 1800B can perform the function of a computing node in the embodiment of the present application.

[0203] It should be understood that Fig.18 The functions of the computing device 1800A shown in the figure can also be completed by multiple computing devices 1800. Similarly, the functions of the computing device 1800B can also be completed by multiple computing devices 1800. And the deployment mode of the modules used to implement the data management method in the computing device can also be adjusted according to the application requirements.

[0204] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements the data management method provided in the embodiment of the present application.

[0205] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the data management method provided by the embodiment of the present application.

[0206] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0207] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and executable code involved in this application are all obtained with full authorization.

[0208] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means a plurality, unless otherwise expressly defined.

[0209] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0210] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0211] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0212] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data management method, characterized in that: The method is applied to the file system layer, and the method includes: Obtain a first data block in a first file to be written into an external memory; Writing the first data block into the external memory and adding a first non-deduplication mark to the first file; Based on the first non-deduplicated mark, an offline deduplication operation is performed on the first data block.

2. The method according to claim 1, characterized in that After writing the first data block into the external memory, the method further includes: A second non-deduplicated mark is added to the first data block.

3. The method according to claim 2, characterized in that The performing an offline deduplication operation on the first data block based on the first non-deduplication mark includes: Based on the first non-deduplication mark, determining the first file from a plurality of files written into the external memory; Based on the second non-deduplication mark, determining the first data block from a plurality of data blocks in the first file; An offline deduplication operation is performed on the first data block.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: receiving a deduplication instruction sent by a user, wherein the deduplication instruction instructs to perform a deduplication operation on a specified range of the first file; The performing an offline deduplication operation on the first data block based on the first non-deduplication mark includes: Based on the deduplication instruction, the designated range is determined in a plurality of files marked with the first non-deduplication mark, and an offline deduplication operation is performed on the data blocks included in the designated range.

5. The method according to claim 4, characterized in that The specified range is the first file, a file set to which the first file belongs, a directory of the first file, or a file cluster of the first file.

6. The method according to any one of claims 1 to 5, characterized in that: The performing an offline deduplication operation on the first data block based on the first non-deduplication mark includes: Determine the first data block based on the first non-deduplication mark; Matching the fingerprint information of the first data block with fingerprint information stored in at least one of the internal memory and the external memory; When the fingerprint information of the first data block matches the stored fingerprint information, performing deduplication processing on the first data block; When the fingerprint information of the first data block does not match the stored fingerprint information, the key value of the first data block is updated to the external memory based on the fingerprint information of the first data block.

7. The method according to claim 6, characterized in that The matching of the fingerprint information of the first data block with the fingerprint information stored in at least one of the internal memory and the external memory includes: Matching the fingerprint information of the first data block with the fingerprint information stored in the memory; When the fingerprint information stored in the memory does not match the fingerprint information of the first data block, the fingerprint information of the first data block continues to be matched with the fingerprint information stored in the external memory.

8. The method according to claim 6, characterized in that The external memory includes a first-level storage unit and a second-level storage unit; The entries stored in the first-level storage unit are entries removed from the memory; The entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: receiving a deletion instruction instructing to delete the second data block; Obtaining a storage location of the key value of the second data block; When the key value of the second data block is stored in the memory and is not stored in the external memory, deleting the key value of the second data block in the memory; When the key value of the second data block is not stored in the internal memory but is stored in the external memory, recording a deletion event indicating deletion of the key value of the second data block, and executing the deletion event at a specified time to delete the key value of the second data block in the external memory; When the key value of the second data block is stored in the internal memory and in the external memory, the key value of the second data block is deleted in the internal memory and the external memory.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Determining whether the first data block is suitable for performing an online deduplication operation; The step of writing the first data block into the external memory and adding a first non-deduplication mark to the first file includes: When the first data block is not suitable for performing an online deduplication operation, the first data block is written into the external memory, and the first non-deduplicated mark is added to the first file.

11. A data management device, characterized in that: The device is applied to the file system layer, and the device includes: An acquisition module, used for acquiring a first data block in a first file to be written into an external memory; A writing module, used for writing the first data block into the external memory and adding a first non-deduplication mark to the first file; A deduplication module is used to perform an offline deduplication operation on the first data block based on the first non-deduplicated mark.

12. The device according to claim 11, characterized in that The writing module is further used to add a second non-deduplicated mark to the first data block.

13. The device according to claim 12, characterized in that The deduplication module is specifically used for: Based on the first non-deduplication mark, determining the first file from a plurality of files written into the external memory; Based on the second non-deduplication mark, determining the first data block from a plurality of data blocks in the first file; An offline deduplication operation is performed on the first data block.

14. The device according to any one of claims 11 to 13, characterized in that: The device also includes: An interactive module, configured to receive a deduplication instruction sent by a user, wherein the deduplication instruction indicates to perform a deduplication operation on a specified range of the first file; The deduplication module is specifically configured to determine the specified range in a plurality of files marked with the first non-deduplication mark based on the deduplication instruction, and perform an offline deduplication operation on the data blocks included in the specified range.

15. The device according to claim 14, characterized in that The specified range is the first file, a file set to which the first file belongs, a directory of the first file, or a file cluster of the first file.

16. The device according to any one of claims 11 to 15, characterized in that The deduplication module is specifically used for: Determine the first data block based on the first non-deduplication mark; Matching the fingerprint information of the first data block with fingerprint information stored in at least one of the internal memory and the external memory; When the fingerprint information of the first data block matches the stored fingerprint information, performing deduplication processing on the first data block; When the fingerprint information of the first data block does not match the stored fingerprint information, the key value of the first data block is updated to the external memory based on the fingerprint information of the first data block.

17. The device according to claim 16, characterized in that The deduplication module is specifically used for: Matching the fingerprint information of the first data block with the fingerprint information stored in the memory; When the fingerprint information stored in the memory does not match the fingerprint information of the first data block, the fingerprint information of the first data block continues to be matched with the fingerprint information stored in the external memory.

18. The device according to claim 16, characterized in that The external memory includes a first-level storage unit and a second-level storage unit; The entries stored in the first-level storage unit are entries removed from the memory; The entries stored in the second-level storage unit are entries removed from the first-level storage unit and processed, and the processing includes compression and / or merging.

19. The device according to any one of claims 11 to 18, characterized in that The device also includes: An interaction module, configured to receive a deletion instruction instructing to delete the second data block; The acquisition module is used to acquire the storage location of the key value of the second data block; a deleting module, configured to delete the key value of the second data block in the memory when the key value of the second data block is stored in the memory and is not stored in the external memory; The deletion module is further configured to, when the key value of the second data block is not stored in the internal memory but is stored in the external memory, record a deletion event indicating deletion of the key value of the second data block, and execute the deletion event at a specified time to delete the key value of the second data block in the external memory; The deleting module is further configured to delete the key value of the second data block in the memory and the external memory when the key value of the second data block is stored in the memory and in the external memory.

20. The device according to any one of claims 11 to 19, characterized in that The device also includes: A judging module, used for judging whether the first data block is suitable for performing an online deduplication operation; The writing module is specifically used to write the first data block into the external memory and add the first non-deduplicated mark to the first file when the first data block is not suitable for performing an online deduplication operation.

21. A computing device cluster, characterized in that: The method comprises at least one computing device, each computing device comprises a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images