Data management method and device

Abstract

The invention discloses a data management method and device, and belongs to the technical field of file systems. The method comprises the following steps: acquiring to-be-written data in a file to be written into an external memory; when the to-be-written data is not duplicated data, allocating a virtual address and a physical address to the to-be-written data, recording the to-be-written data in the physical address, and recording a corresponding relationship between the virtual address and the physical address and a corresponding relationship between the virtual address and a logic address of the to-be-written data; the logic address is used for indicating the position of the to-be-written data in the file; when the to-be-written data is duplicated data, acquiring a virtual address of the duplicated data of the to-be-written data, and recording a corresponding relationship between the virtual address and a logic address of the to-be-written data; according to the invention, direct mapping between the logic address and the physical address is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of file systems, and in particular to a data management method and device thereof. Background Art

[0002] Data deduplication technology is also called de-duplication technology. Its principle is to reduce redundant data by retaining only one physical block copy for logical data blocks with the same data and indexing the access to these logical blocks to this physical block. Data deduplication technology can not only reduce real-time redundant writes on the critical path and extend the service life of the equipment, but also save storage capacity and achieve efficient reuse of storage space. Current data deduplication generally divides files into data blocks, and then compares the data blocks according to the fingerprint information of the data blocks, and references the logical files with the same logical data content to the only shared physical copy to eliminate redundant data.

[0003] Currently, when the data block to be written is duplicate data, the file system layer directly saves the physical address of the copy in the parent node of the duplicate data and updates the reference count of the data block in the metadata. When the data block to be written is not duplicate data, the file system layer allocates a new physical address and physical block to the data block and saves the fingerprint information of the data block for subsequent block deduplication. Among them, the parent node is used to store the mapping relationship between the data and the physical address of the data.

[0004] However, this deduplication method will result in a many-to-one mapping of logical addresses to physical addresses. This will cause two problems: First, it is complex and difficult to organize a one-to-many mapping in a metadata area with limited space. Second, during the garbage collection process, if a physical block corresponds to multiple logical blocks, it is necessary to traverse all parent nodes and modify the address information therein, which will undoubtedly slow down the processing speed of garbage collection. Summary of the invention

[0005] The present application provides a data management method and device thereof. The present application avoids direct mapping of logical addresses to physical addresses. The technical solution provided by the present application is as follows:

[0006] In a first aspect, the present application provides a data management method. The method is applied to the file system layer. The method includes: obtaining data to be written in a file to be written to an external storage device; when the data to be written is not duplicate data, allocating a virtual address and a physical address to the data to be written, recording the data to be written in the physical address, recording the correspondence between the virtual address and the physical address, and the correspondence between the virtual address and the logical address of the data to be written, the logical address is used to indicate the position of the data to be written in the file; when the data to be written is duplicate data, obtaining the virtual address of the duplicate data of the data to be written, and recording the correspondence between the virtual address and the logical address of the data to be written.

[0007] In the data management method, the file system layer allocates virtual addresses to data, and records the correspondence between virtual addresses and physical addresses, as well as the correspondence between virtual addresses and logical addresses, so that the virtual addresses can isolate logical addresses and physical addresses, avoiding direct mapping of logical addresses to physical addresses. In addition, since the virtual addresses of duplicate data are the same, the virtual addresses and physical addresses can form a one-to-one mapping, which reduces the complexity of the file system mapping relationship, helps to improve the processing speed of garbage collection, and ensures the read and write performance of the file system.

[0008] Optionally, since the data written by the file system layer may be duplicate data or may not be duplicate data, in order to manage the consistency of the written data, the file system layer may also record or update the total number of logical addresses corresponding to the virtual address each time the data is written. For example, the method further includes: when the data to be written is not duplicate data, recording the total number of logical addresses corresponding to the virtual address; when the data to be written is duplicate data, updating the total number of logical addresses corresponding to the virtual address.

[0009] In a possible implementation, the method further includes: obtaining a first physical address of the data to be migrated in the external memory; based on the first physical address, obtaining a virtual address of the data to be migrated; writing the data to be migrated to a second physical address; and based on the second physical address, updating the correspondence between the virtual address and the physical address of the data to be migrated.

[0010] As can be seen from the above, when migrating data, what needs to be modified is the correspondence between the virtual address and the physical address, and there is no need to modify the correspondence between the logical address and the physical address as in the related art. In addition, since the virtual addresses of duplicate data are the same, the virtual address and the physical address can form a one-to-one mapping. When modifying the correspondence between the virtual address and the physical address, no matter whether the data to be migrated is unique data or duplicate data, only the virtual address needs to be modified. There is no need to modify multiple logical addresses corresponding to the duplicate data as in the related art. Therefore, the data migration speed can be guaranteed and the performance of the file system can be guaranteed.

[0011] Optionally, before writing the data to be migrated to the second physical address, the total number of logical addresses corresponding to the virtual address of the data to be migrated can be obtained, and based on the total number, it is determined whether to execute the process of writing the data to be migrated to the second physical address. If the total number is 0, it means that all logical references to the data to be migrated have become invalid during the migration, and the data to be migrated has become invalid data, and there is no need to migrate the data to be migrated. If the total number is not 0, it means that the data to be migrated is still valid data, and it is necessary to migrate the data to be migrated. Therefore, when the total number of logical addresses corresponding to the virtual address of the data to be migrated is not 0, the data to be migrated is written to the second physical address. When the total number of logical addresses corresponding to the virtual address of the data to be migrated is 0, the migration operation of the data to be migrated is stopped, and the data to be migrated can be discarded at this time. In this way, the amount of data migration can be reduced, which helps to shorten the time consumption of data migration and helps to reduce the time consumed by garbage collection. In order to manage data consistency, after the data to be migrated is read into the memory, the reference count of the data to be migrated can be increased by one, and before executing the operation of writing the data to be migrated to the second physical address, the reference count of the data to be migrated can be reduced by one, and then it is determined whether the reference count of the data to be migrated is 0.

[0012] In one implementation, the data to be migrated is valid data stored in the physical block to be reclaimed.

[0013] In one implementation, obtaining the virtual address of the duplicate data of the data to be written includes: obtaining the virtual address of the duplicate data of the data to be written based on a correspondence between query data of the data to be written and the virtual address.

[0014] Optionally, the method also includes: obtaining fingerprint information of the data to be written; matching the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory; when the fingerprint information of the data to be written matches the fingerprint information of the written data, determining that the data to be written is duplicate data; when the fingerprint information of the data to be written does not match the fingerprint information of the written data, determining that the data to be written is not duplicate data.

[0015] Optionally, the memory stores the fingerprint information of the written data in descending order of access popularity of the written data. By managing the fingerprint information according to access popularity, when writing data, it is possible to quickly find recently written data in the memory and remove long-term unused data.

[0016] In the second aspect, the present application provides a data management device. The device is applied to the file system layer, and the device includes: an acquisition module, which is used to acquire the data to be written in the file to be written to the external storage device; a processing module, which is used to allocate a virtual address and a physical address to the data to be written when the data to be written is not duplicate data, record the data to be written in the physical address, record the correspondence between the virtual address and the physical address, and the correspondence between the virtual address and the logical address of the data to be written, and the logical address is used to indicate the position of the data to be written in the file; the processing module is also used to acquire the virtual address of the duplicate data of the data to be written when the data to be written is duplicate data, and record the correspondence between the virtual address and the logical address of the data to be written.

[0017] Optionally, the processing module is further used to: when the data to be written is not duplicate data, record the total number of logical addresses corresponding to the virtual address; when the data to be written is duplicate data, update the total number of logical addresses corresponding to the virtual address.

[0018] Optionally, the acquisition module is also used to acquire the first physical address of the data to be migrated in the external memory; the acquisition module is also used to acquire the virtual address of the data to be migrated based on the first physical address; the processing module is also used to write the data to be migrated to the second physical address; the processing module is also used to update the correspondence between the virtual address and the physical address of the data to be migrated based on the second physical address.

[0019] Optionally, the processing module is specifically configured to: when the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is not 0, write the data to be migrated into the second physical address.

[0020] Optionally, the processing module is further configured to: when the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is 0, stop the migration operation on the data to be migrated.

[0021] Optionally, the data to be migrated is valid data stored in the physical block to be reclaimed.

[0022] Optionally, the processing module is specifically used to obtain the virtual address of the duplicate data of the data to be written based on the corresponding relationship between the data to be written and the virtual address.

[0023] Optionally, the processing module is also used to: obtain fingerprint information of the data to be written; match the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory; when the fingerprint information of the data to be written matches the fingerprint information of the written data, determine that the data to be written is duplicate data; when the fingerprint information of the data to be written does not match the fingerprint information of the written data, determine that the data to be written is not duplicate data.

[0024] Optionally, the memory stores fingerprint information of the written data in descending order of access popularity of the written data.

[0025] In a third aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method provided in the first aspect of the present application and any possible implementation thereof.

[0026] In a fourth aspect, the present application provides a computing device cluster, including multiple computing devices, the multiple computing devices including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors running the program instructions, so that the computing device cluster executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0028] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of an operating system I / O stack provided in an embodiment of the present application;

[0030] Figure 2 It is a process diagram of a DMdedup deduplication implementation method provided in an embodiment of the present application;

[0031] Figure 3 This is a method provided by the embodiment of the present application. Figure 2 Flowchart of the deduplication process shown;

[0032] Figure 4 It is a process diagram of a SmartDedup deduplication implementation method provided in an embodiment of the present application;

[0033] Figure 5 This is a method provided by the embodiment of the present application. Figure 4 Flowchart of the deduplication process shown;

[0034] Figure 6 It is a schematic diagram of an implementation environment involved in a data management method provided in an embodiment of the present application;

[0035] Figure 7 is a schematic diagram of an implementation environment involved in another data management method provided in an embodiment of the present application;

[0036] Figure 8 is a schematic diagram of a data writing process provided by an embodiment of the present application;

[0037] Fig. 9 This is a method provided by the embodiment of the present application. Figure 8 A flowchart of the process shown;

[0038] Fig.10 This is a schematic diagram of determining whether the data to be written is duplicate data provided by an embodiment of the present application;

[0039] Fig.11 It is a schematic diagram of memory management fingerprint information provided by an embodiment of the present application;

[0040] Fig.12 is a flowchart of a garbage collection process provided by an embodiment of the present application;

[0041] Fig.13 is a process schematic diagram of a garbage collection process provided by an embodiment of the present application;

[0042] Fig.14 It is a schematic diagram of a metadata layout provided in an embodiment of the present application;

[0043] Fig.15 is a schematic diagram of a data management device provided in an embodiment of the present application;

[0044] Fig.16 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0045] Fig.17 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0047] To facilitate understanding, the technology and background involved in the embodiments of the present application are first explained below.

[0048] The log-structured file system (LFS) has the features of out-of-place update and garbage collection. Compared with traditional jorunaling file systems (such as EXT and XFS) and copy-on-write file systems (such as BtrFS and ZFS), it has better random write and small-granularity transaction update performance.

[0049] The flash-friendly file system (F2FS) is a log-structured file system designed for flash memory devices and is widely used in smart terminal scenarios such as mobile phones and high-speed servers. The flash-friendly file system organizes the storage area into a log structure, writes data such as user data, file mappings, and directory indexes in the form of logs, and converts random writes of data and file structures into sequential writes, thereby improving the overall write performance of the file system, slowing down the wear of devices with limited lifespans (such as flash memory), and has an efficient file fragment recovery mechanism. Currently, F2FS has been widely used in smart terminal devices (such as mobile phones and smart TVs) and servers, and these devices also have a large amount of duplicate data from application caches and backups, user duplicate files, and system updates and snapshots.

[0050] Logical / virtual / physical address (LA / VA / PA) is the three-layer address used in this application. The logical address refers to the offset of the data block relative to the file, that is, the position of the data block in the file. The physical address refers to the offset of the data block in the entire file system address space, that is, the actual location of the data in the physical storage space. The virtual address is used to isolate the two to reduce the complexity of deduplication file system mapping.

[0051] Garbage collection (GC) is a basic operation in F2FS. Since F2FS organizes the storage area into a log structure, when the remaining log free segments are insufficient, it is necessary to migrate the valid data in the old space and reclaim the old space. The migration of valid data requires garbage collection to modify the address information in the parent node of the data block, so the reverse mapping from physical address to logical address is required.

[0052] Node address table (NAT): In native F2FS, corresponding node IDs are assigned to nodes, and NAT can provide conversion from node IDs to node physical addresses.

[0053] The virtual address table (VAT) is a metadata structure used to convert virtual addresses to physical addresses, and plays an important role in file reading and writing. Some hot items will be cached in memory.

[0054] The segment summary area or reverse mapping table (SSA) is a metadata structure in F2FS that stores the reverse mapping from physical addresses to logical addresses (i.e., parent node related information), so it is also called the reverse mapping table, which is mainly used during garbage collection. In this application, since the virtual address layer is added, the information stored in the SSA is changed to the mapping from physical addresses to virtual addresses, which is also mainly used in the garbage collection process.

[0055] The segment information table (SIT) is a structure used by F2FS to centrally manage segment status. Its main function is to maintain segment allocation information.

[0056] The reference count (RC) is used to mark the number of times a copy is referenced in the deduplication system, that is, how many logical blocks reference a physical block.

[0057] Device mapper is a framework provided by the Linux kernel for mapping physical block devices to advanced virtual block devices. It is the basis of the logical volume manager.

[0058] The page cache is created by the operating system using memory and is an area used to cache data pages in external storage (such as a disk or solid-state drive).

[0059] Direct I / O refers to reading and writing external memory directly. The upper-level file system or application only records the file mapping or address index, rather than directly accessing the data content in the memory, without going through the page cache. Therefore, for data written through Direct I / O, additional reading is required to obtain the data content.

[0060] An inode is a data structure used to describe objects such as files and directories.

[0061] The super block is used to store the metadata of the file system. The super block contains information such as the file system type, size, status, and usage. Its function is to enable the operating system to identify and manage the file system.

[0062] A checkpoint is used to represent the consistent state of a system. In a file system, the consistent state of the system is saved to a persistent storage so that it can be restored based on the consistent state in the event of a failure.

[0063] Data deduplication technology is also called de-duplication technology. Its principle is to reduce redundant data by retaining only one physical block copy for logical data blocks with the same data and indexing the access to these logical blocks to this physical block. Data deduplication technology can not only reduce real-time redundant writes on the critical path and extend the service life of the equipment, but also save storage capacity and achieve efficient reuse of storage space. Current data deduplication generally divides files into data blocks, and then compares the data blocks according to the fingerprint information of the data blocks, and references the logical files with the same logical data content to the only shared physical copy to eliminate redundant data.

[0064] At present. According to the triggering time, deduplication is generally divided into online deduplication and offline deduplication. Online deduplication is performed before the data is written to the disk, while offline deduplication is to retain the original data in the external memory first, and then trigger offline deduplication under specific conditions. The data management method provided in the embodiment of the present application mainly relates to online deduplication. The advantage of online deduplication is that since the deduplication process is completed before the data is written to the disk, the amount of data written can be reduced and redundant writes can be eliminated in real time. However, its disadvantage is that if the fingerprint (FP) information is stored in the memory, it is usually impossible to retain all fingerprint information, resulting in the inability to perform thorough deduplication. If stored in an external memory, since the external memory needs to be repeatedly read and written on the critical path of the data reading and writing process, the input / output (I / O) speed of the system will be significantly affected. This results in the problems of high CPU overhead, long tail delay and high memory usage in the write path of online deduplication. Offline deduplication aims to write data to the storage device first, and then eliminate duplicate data when the system is idle, and the deduplication benefits cannot be obtained immediately.

[0065] Deduplication can be done at different levels of the operating system I / O stack. Figure 1 Schematic diagram of an operating system I / O stack provided in an embodiment of the present application. Figure 1As shown, the operating system I / O stack includes: a virtual file system (VFS) layer, a file system (FS) layer, a block layer, and a hard disk driver layer. In some implementations, the hard disk driver layer can be a small computer system interface (SCSI) layer. Among them, the virtual file system is responsible for the general file abstract semantics, abstracts a unified operation interface for the upper layer, and switches different file systems on the I / O path. The file system is responsible for abstracting the concept of "file" and maintaining the location mapping of "file" data to the block layer. The block layer is used to make a unified abstraction of the underlying hardware devices, and is mainly responsible for the implementation of the I / O scheduling strategy. For example, the block layer collects batch I / Os based on the scheduling strategy and aggregates and sends them down, merges I / O requests to reduce the number of I / Os, etc. The hard disk driver layer is responsible for docking with the hardware disk.

[0066] Currently, deduplication at the file system layer and block layer is a common processing method.

[0067] DMdedup is a typical example of deduplication at the block layer. DMdedup uses the device mapper technology in the kernel to map two devices into a logical volume using a priori duplication rates. After the data is written to the logical volume, deduplication is completed at the block layer. The deduplicated data and the metadata generated by deduplication are stored separately on two devices, which are called data devices and metadata devices. When writing data, the metadata device is first accessed based on the fingerprint information of the data block, and then the fingerprint information is compared with the fingerprint information of the stored data. Based on the comparison results, duplicate data is deduplicated or non-duplicate data is stored in the data device, and the metadata is modified to save relevant information. To read data, the metadata device must also be accessed to obtain address information before reading from the data device.

[0068] Figure 2 This is a process diagram of DMdedup's deduplication implementation method. Figure 3 Here, we take the case where both node 1 and node 2 write a data block carrying data "2A" as an example. Figure 2 and Figure 3 As shown, the writing process mainly includes the following five steps:

[0069] 1.DMdedup creates a logical volume in the block layer using physical device 1 and physical device 2.

[0070] 2. The file system layer perceives the logical volume and formats the file system on the logical volume. When receiving a write data request from the upper layer, the file system directly passes the data request downward without performing deduplication.

[0071] 3. After the logical volume in the block layer obtains the data block "2A" to be written, its fingerprint information is compared with the fingerprint information of the written data, and whether the data block "2A" is duplicate data is determined based on the comparison result.

[0072] 4. If data block "2A" is duplicate data, the block layer maps data block "2A" to the actual address of data block "2A" written in the data device, and updates the metadata in the metadata device. If data block "2A" is not duplicate data, the block layer writes data block "2A" to the data device normally and adds relevant metadata in the metadata device.

[0073] 5. The block layer writes the only copy to physical device 1 based on the data type, while the metadata generated by deduplication is stored separately in physical device 2.

[0074] However, DMdedup has performance and "false storage space" issues. Due to deduplication, metadata is stored on a separate device in the external storage. In the process of reading and writing data, it is necessary to first access the metadata device on the critical path to obtain relevant address information before operations can be performed on the data device, resulting in serious performance degradation. At the same time, because the logical volume is generated using a priori repetition rate, its size is fixed, and the file system cannot perceive the deduplication semantics, a "false storage space" problem will arise. In other words, if the priori repetition rate used is higher than the repetition rate of the actual load, the file system will think that there is still room to store data, but the storage space in the actual device is full. Conversely, the actual storage space in the device cannot be fully utilized.

[0075] SmartDedup is a typical example of deduplication in the file system layer. SmartDedup focuses on lightweight online / offline deduplication combined with technical solutions in resource-constrained mobile devices. In terms of mapping relationships, SmartDedup allows the same physical address to be stored in the parent node of a file data block, and deduplication is accomplished by directly mapping the logical address in the file system to the physical address.

[0076] Figure 4 This is a process diagram of SmartDedup's deduplication implementation method. Figure 5 Here, we also take the case where both node 1 and node 2 write a data block carrying data "2A" as an example. Figure 4 and Figure 5 As shown, the writing process mainly includes the following five steps:

[0077] 1. The file system layer obtains the data block to be written and calculates the fingerprint information of the data block.

[0078] 2. The file system layer performs comparison based on fingerprint information.

[0079] 3. If the comparison result indicates that the data block is duplicate data, the file system layer directly saves the physical address of the copy in the parent node of the duplicate data, for example, storing the physical address pointing to the data block "2A" in node 1 in node 2, and updating the reference count of the data block in the metadata. If the comparison result indicates that the data block is not duplicate data, the file system layer allocates a new physical address and physical block to the data block, and saves the fingerprint information of the data block for subsequent block deduplication. Among them, the parent node is used to store the mapping relationship between data and the physical address of the data.

[0080] 4. Write the non-duplicate data normally, such as writing the data block to the physical block, and setting the address of the physical block to the assigned physical address. Since the duplicate data already has a copy in the storage device, the file system layer terminates the actual writing process of the duplicate data.

[0081] 5. Complete writing.

[0082] However, SmartDedup believes that the file system layer has saved complete mapping information. In file systems like F2FS that require active garbage collection, since the garbage collection operation requires relevant information of the parent node, SmartDedup does not consider this situation. Due to deduplication, SmartDedup will cause a many-to-one mapping of logical addresses to physical addresses. At this time, if garbage collection needs to be completed, a one-to-many reverse mapping of physical addresses to logical addresses must be retained. This will cause two problems: First, it is complex and difficult to organize a one-to-many mapping in a metadata area with limited space. Second, during the garbage collection process, if a physical block corresponds to multiple logical blocks, that is, there are multiple parent nodes, it is necessary to traverse all parent nodes and modify the address information therein, which will undoubtedly slow down the processing speed of garbage collection.

[0083] Based on this, an embodiment of the present application provides a data management method. The method is applied to the file system layer. In the data management method, the file system layer obtains the data to be written in the file to be written to the external storage. When the data to be written is not duplicate data, a virtual address and a physical address are allocated to the data to be written, and the data to be written is recorded in the physical address, and the correspondence between the virtual address and the physical address, and the correspondence between the virtual address and the logical address of the data to be written are recorded. When the data to be written is duplicate data, the virtual address of the duplicate data of the data to be written is obtained, and the correspondence between the virtual address and the logical address of the data to be written is recorded. Among them, the logical address is used to indicate the position of the data to be written in the file.

[0084] In the data management method, the file system layer allocates virtual addresses to data, and records the correspondence between virtual addresses and physical addresses, as well as the correspondence between virtual addresses and logical addresses, so that the virtual addresses can isolate logical addresses and physical addresses, avoiding direct mapping of logical addresses to physical addresses. In addition, since the virtual addresses of duplicate data are the same, the virtual addresses and physical addresses can form a one-to-one mapping, which reduces the complexity of the file system mapping relationship, helps to improve the processing speed of garbage collection, and ensures the read and write performance of the file system.

[0085] The following first describes an implementation environment involved in a data management method provided in an embodiment of the present application.

[0086] Figure 6 Schematic diagram of an implementation environment involved in a data management method provided in an embodiment of the present application. Figure 6 As shown, the implementation environment includes: a computing device 10. The computing device 10 can obtain data to be written into a file in an external memory and write the data into the external memory. Figure 6 In the illustrated implementation environment, the file to be written may be a file generated by the computing device 10 itself, or a file received by the computing device 10 from the outside.

[0087] In one implementation, the data management method provided in the embodiment of the present application can be implemented by running an executable program on the computing device 10. For example, the executable program of the data management method can be presented in the form of an application installation package. After the application installation package is installed in the computing device 10, the data management method can be implemented by running the executable program. In this case, the computing device 10 can be a terminal. The terminal can be a computer, a personal computer, a portable mobile terminal, a multimedia player, an e-book reader, or a wearable device, etc.

[0088] Figure 7 FIG. 1 is a schematic diagram of an implementation environment involved in another data management method provided in an embodiment of the present application. Figure 7 As shown, the implementation environment may include: a client 20 and a computing device 10. The computing device 10 is used to store data and execute the data management method provided in the embodiment of the present application. The client 20 can establish a communication connection with the computing device 10. For example, a communication connection can be established between the client 20 and the computing device 10 through a network. Optionally, the network can be a local area network, the Internet, or other networks, which is not limited in the embodiment of the present application.

[0089] The client 20 is used for the user to interact with the computing device 10. In one implementation, the client 20 is used to send instructions to the computing device 10 according to the user's instructions. For example, the client 20 is used to send a data write instruction to the computing device 10 according to the user's instructions to instruct the writing of data to the external memory. For another example, the client 20 is used to send a data read instruction to the computing device 10 according to the user's instructions to instruct the reading of data from the external memory. The computing device 10 is used to receive the instruction sent by the client 20 and perform the operation indicated by the instruction. For example, the computing device 10 is used to receive a data write instruction, and according to the data management method provided in the embodiment of the present application, the data to be written is stored in the external memory based on the data write instruction. For another example, the computing device 10 is used to receive a data read instruction, and according to the data management method provided in the embodiment of the present application, the target data is obtained based on the data read instruction, and the target data is fed back to the client 20.

[0090] In one implementable manner, the client 20 may be a desktop computer, a laptop computer, a mobile phone, a smart phone, a tablet computer, a multimedia player, a smart home appliance, an artificial intelligence device, a smart wearable device, an e-reader, a smart vehicle-mounted device, or an Internet of Things device. The computing device 10 may be a server (such as a cloud server). Typically, the computing device 10 may be a server cluster consisting of several servers, or may be implemented by a cloud computing service center. Among them, a large number of basic resources owned by a cloud service provider are deployed in the cloud computing service center. For example, computing resources, storage resources, and network resources are deployed in the cloud computing service center. The cloud computing service center can utilize this large amount of basic resources to implement the data management method provided in the embodiment of the present application.

[0091] When the computing device 10 is implemented through a cloud computing service center. The user can access the cloud platform through the client 20, and use the storage service provided by the external memory through the cloud platform. At this time, the functions implemented by the data management method provided in the embodiment of the present application can be abstracted by the cloud service provider into a storage cloud service on the cloud platform, and the cloud platform can use the resources in the cloud computing center to provide the storage cloud service to the user. After the user purchases the storage cloud service on the cloud platform, the storage cloud service can store the data that needs to be written for the user, or provide the stored data to the user. Optionally, the cloud platform can be a cloud platform of a central cloud, a cloud platform of an edge cloud, or a cloud platform including a central cloud and an edge cloud, and the embodiment of the present application does not specifically limit it. It should be noted that in Figure 7 In the implementation environment shown, the computing device 10 can also be implemented by other resource platforms besides the cloud platform, which is not specifically limited in the present embodiment. In this case, the computing device 10 can be implemented by resources in other resource platforms and provide relevant storage services to users.

[0092] In one implementation, the data management method provided by the embodiment of the present application can be applied to a computing device deployed with F2FS. Due to F2FS's requirement for sequential writing, the segments in the old space may contain both valid data and data that has become invalid due to deletion or update. In this case, it is necessary to first perform garbage collection (GC) on the old space and migrate the valid data until all data in the segments of the old space become invalid, and then reclaim the old space. Among them, the file system manages the data of the file system in units such as blocks, segments, sections, zones, and file systems. The granularity of blocks, segments, sections, zones, and file systems increases in sequence.

[0093] It should be understood that the above content is an exemplary description of the application scenario of the data management method provided in the embodiment of the present application, and does not constitute a limitation on the application scenario of the data management method. A person of ordinary skill in the art will know that as business needs change, its application scenario can be adjusted according to application needs. For example, the data management method provided in the embodiment of the present application may not be limited to application to computing devices deployed with F2FS, but may also be applied to other file systems with the same technical problems. In addition, when the data management method provided in the embodiment of the present application is applied to a computing device deployed with F2FS, it inherits the function of deduplication into F2FS, and the file system can be regarded as a flash-friendly deduplication file system (F2DFS) modified on the basis of F2FS.

[0094] The data management method provided in the embodiment of the present application is applied to Figure 7 Taking the application scenario shown as an example, the implementation process of the data management method provided by the embodiment of the present application is described. In the data management method, the file system layer allocates a virtual address to the data during the data writing process. The impact of the virtual address on data management is mainly reflected in the data writing process and the data migration process. The data writing process is first described below.

[0095] Figure 8 It is a schematic diagram of a data writing process provided in an embodiment of the present application. Fig. 9 This is a corresponding flow chart provided by an embodiment of the present application. Similarly, the example of both node 1 and node 2 writing a data block carrying data "2A" is used for explanation. Figure 8 and Fig. 9 As shown, the data writing process includes the following steps:

[0096] Step 801: The file system layer obtains the data to be written in the file to be written into the external storage.

[0097] The file to be written may be a file generated by the computing device itself, or a file received by the computing device from outside. For example, the file to be written may be a file indicated by a write request sent by a user. In one implementation, the file system layer may process the data in the file in units of data blocks, and the data blocks may be obtained by dividing the data in the file. The embodiment of the present application is described by taking the file system layer processing the data blocks of the file as an example.

[0098] Step 802: The file system layer determines whether the data to be written is duplicate data.

[0099] In one implementation, the file system layer may obtain fingerprint information of the data to be written, and determine whether the data to be written is duplicate data based on the fingerprint information of the data to be written. Fig.10 A schematic diagram of determining whether the data to be written is duplicate data is provided in an embodiment of the present application.

[0100] like Fig.10 As shown, the implementation process includes:

[0101] Step 8021: Obtain fingerprint information of the data to be written.

[0102] The fingerprint information of the data to be written is a unique identifier of the data to be written. The fingerprint information of the data to be written can be obtained based on the hash value of the data to be written. For example, the fingerprint information of the data to be written can be obtained by performing a collision-resistant hash calculation on the data to be written. Collision resistance refers to a low probability of duplication of the generated hash value. For example, when writing a file to an external storage system, since the data of the file can be first cached in the page cache of the memory, the hash calculation can be performed in units of pages in the page cache, and the result of the hash calculation is the fingerprint information of the data to be written. Optionally, when performing a hash calculation in units of pages in the page cache, the data stored in the page can also be divided according to a specified size (such as 4 kilobytes (KB)), and the corresponding hash value is obtained based on the data obtained by the division.

[0103] Step 8022: Match the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory.

[0104] After obtaining the fingerprint information of the data to be written, the fingerprint information can be matched with the fingerprint information of the data that has been written. Since the fingerprint information is used to uniquely identify the data, when the fingerprint information of the data to be written matches the fingerprint information of the data that has been written, it can be considered that the data to be written is the same as the data that has been written, and the data to be written is determined to be duplicate data. When the fingerprint information of the data to be written does not match the fingerprint information of the data that has been written, it can be considered that the data to be written is different from the data that has been written, and it is determined that the data to be written is not duplicate data. The index of the data content can be quickly identified through the fingerprint information of the data, and its efficiency is much higher than the method of comparing the byte-level data content, which can ensure the efficiency of judging whether the first data block is a duplicate data block. In addition, since its collision probability (false positive) is extremely low, even much lower than the hardware error probability, it can ensure the accuracy of judging whether the first data block is a duplicate data block.

[0105] In an embodiment of the present application, the fingerprint information of all written data is stored in the memory. After obtaining the fingerprint information of the data to be written, it is possible to search in the memory whether there is the same fingerprint based on the fingerprint information of the data to be written. When there is a fingerprint in the memory that is the same as the fingerprint information of the data to be written, it is determined that the fingerprint information of the data to be written matches the fingerprint information of the written data. When there is a fingerprint in the memory that is different from the fingerprint information of the data to be written, it is determined that the fingerprint information of the data to be written does not match the fingerprint information of the written data. At this time, the fingerprint information of the data to be written can also be written into the memory.

[0106] In one implementation, the memory can manage the fingerprint information according to the prefix of the fingerprint information. For example, the memory maintains multiple memory sets (such as memory buckets), and the multiple memory sets correspond to multiple prefixes respectively, and any memory set is used to store all fingerprint information with corresponding prefixes. Among them, the size of the memory set can be scaled according to the device resources. After obtaining the fingerprint information of the data to be written, it can be determined whether there is a memory set corresponding to the prefix in the memory according to the prefix of the fingerprint information. When the memory set corresponding to the prefix does not exist in the memory, it is determined that the fingerprint information of the data to be written does not match the fingerprint information of the written data. When the memory set corresponding to the prefix exists in the memory, continue to search whether the fingerprint information exists in the memory set. When the fingerprint information does not exist in the memory set, it is determined that the fingerprint information of the data to be written does not match the fingerprint information of the written data. When the fingerprint information exists in the memory set, it is determined that the fingerprint information of the data to be written does not match the fingerprint information of the written data. For example, Fig.11As shown, the memory maintains multiple memory buckets, one of which corresponds to the prefix B452, and the memory bucket is used to store all fingerprint information with the prefix B452. After obtaining the fingerprint information B4522F02F35C5340F620D37AD66434D9D3CEB69C of the data to be written, it can be obtained that the prefix of the fingerprint information is B452. Then, the memory is accessed with the prefix B452 as the key to search for the pointer of the location of the fingerprint bucket, so as to obtain the memory bucket corresponding to B452 in the memory. Then continue to search whether there is fingerprint information B4522F02F35C5340F620D37AD66434D9D3CEB69C in the memory bucket corresponding to B452. Since the fingerprint information exists in the memory bucket shown in 10, it is determined that the fingerprint information of the data to be written is the same as the fingerprint information of the data that has been written, and the data to be written is duplicate data. Among them, the prefix of the fingerprint information can be represented by the characters of the specified number of digits from left to right in the fingerprint information. The designated number of bits can be set according to application requirements, for example, the designated number of bits is four.

[0107] It should be noted that the memory can also manage the fingerprint information according to the access heat of the written data. For example, the memory stores the fingerprint information of the written data in order from high to low access heat of the written data. Then each time the fingerprint information of the data to be written matches the fingerprint information of the written data, the order of the fingerprint information in the memory can be adjusted accordingly. For example, when the fingerprint information of the data to be written matches the fingerprint information of the written data, it can be considered that the fingerprint information has the highest current heat, and the fingerprint information can be moved to the head of the memory bucket. In addition, when the memory bucket is full, the fingerprint information at the end of the memory bucket can be deleted. By managing the fingerprint information according to the access heat, it is possible to quickly find recently written data and remove long-term unused data in the memory when writing data.

[0108] Step 8023: When the fingerprint information of the data to be written matches the fingerprint information of the data already written, it is determined that the data to be written is duplicate data.

[0109] Step 8024: When the fingerprint information of the data to be written does not match the fingerprint information of the data already written, it is determined that the data to be written is not duplicate data.

[0110] It should be noted that before executing step 802, it can be determined whether the data to be written needs to be deduplicated online. When the data to be written needs to be deduplicated online, step 802 is executed again. When the data to be written does not need to be deduplicated online, the data to be written is written according to the writing process of step 803. In one implementation, it can be determined whether it is necessary to perform online deduplication on the data to be written according to the current write mode and the load of the file system. For example, when the process of performing the write operation in the write mode is incompatible with the execution process of deduplication, there is no need to perform online deduplication on the data to be written. For example, when the write mode writes data by directly flushing data (such as naked input and output (Direct IO)), since the writing method is incompatible with the execution process of deduplication, there is no need to perform online deduplication on the data to be written in the write mode. The read and write process of Direct IO does not pass through the page cache, and directly reads and writes to the external memory. For example, when the overhead of performing online deduplication on the data to be written in the write mode does not match the benefit, there is no need to perform online deduplication on the data to be written. For example, when performing online deduplication on data to be written in write mode will introduce a lot of additional overhead, but the overhead is far less than the benefits brought by online deduplication, there is no need to perform online deduplication on the data to be written in write mode.

[0111] Step 803: When the data to be written is not duplicate data, the file system layer allocates a virtual address and a physical address for the data to be written, records the data to be written in the physical address, records the correspondence between the virtual address and the physical address, and the correspondence between the virtual address and the logical address of the data to be written, and the logical address is used to indicate the position of the data to be written in the file.

[0112] When the data to be written is not duplicate data, the file system layer needs to write the data to be written into the external memory in accordance with the normal writing process. The implementation process includes: the file system layer allocates a virtual address and a physical address to the data to be written, then writes the data to be written into the physical block indicated by the physical address (also called writing the data to be written into the physical address), and improves the writing process according to the writing result. Among them, improving the writing process according to the writing result mainly includes recording the relevant information of the writing event. For example, recording the correspondence between the virtual address and the physical address of the data to be written, the correspondence between the virtual address and the logical address, and updating the relevant metadata and other information. The virtual address serves as a mapping bridge between the physical address and the logical address, and makes the physical address and the logical address have no direct mapping relationship, which plays a role in isolating the logical address and the physical address. The data corresponds to the virtual address one by one, and accordingly, the physical block of the data corresponds to the virtual address one by one. Optionally, since the data written by the file system layer may be duplicate data or not, in order to manage the consistency of the written data, the file system layer can also record or update the total number of logical addresses corresponding to the virtual address each time the data is written. For example, when the data to be written is not duplicate data, a new record item of the total number of logical addresses corresponding to the virtual address can be created, and the total number can be initialized to 1. In the embodiment of the present application, since the virtual address corresponds to the physical address one-to-one, the total number of logical addresses corresponding to the virtual address is also the total number of logical addresses corresponding to the physical address. Therefore, the total number is also called the number of times the physical block indicated by the physical address is referenced by the logical address, that is, the total number is also called the reference count of the data to be written.

[0113] In one implementation, the file system layer may first allocate a virtual address for the data to be written, and then allocate a physical address for the virtual address. When allocating a virtual address, one of the unused virtual addresses may be allocated to the data to be written. Alternatively, in a scenario where data is updated, a virtual address that needs to be updated (hereinafter referred to as the old VA) may be allocated to the data to be written. Furthermore, when the usage of the old VA is different, the implementation details of allocating the old VA may be slightly different. For example, when the total number of logical addresses corresponding to the old VA is 1, since the old VA itself needs to be refreshed, the old VA may be reused, that is, without first recycling the old VA, the old VA may be directly allocated to the data to be written, and the physical address corresponding to the old VA may be allocated to the data to be written. After the data to be written is written in the physical address based on the old VA, since the total number of logical addresses corresponding to the old VA is 1, there is no need to update the total number of logical addresses corresponding to the old VA to 1. When the total number of logical addresses corresponding to the old VA is greater than 1, the total number of logical addresses corresponding to the old VA needs to be reduced by one to indicate that the reference count of the physical address corresponding to the old VA is reduced by one. Therefore, reusing the old VA can further improve the writing efficiency of the data to be written. Among them, the scenario of updating data refers to the process of deleting the stored data and rewriting the new data in the same logical address. It should be noted that the above-mentioned method of reusing the old VA is applicable to the scenario where the rewritten data is not duplicate data. In the scenario where the rewritten data is duplicate data, the virtual address of the duplicate data is recorded on its logical address, not the newly allocated virtual address. At this time, no matter how much the reference count corresponding to the old VA is, it needs to be reduced by one accordingly. If the original reference count of the old VA is 1, it will become 0 after being reduced by one. At this time, this virtual address can be recycled for subsequent writing. If the original reference count of the old VA is greater than 1, it will not be 0 after being reduced by one, indicating that this virtual address is still referenced by at least one logical address and does not need to be recycled temporarily.

[0114] Optionally, the memory not only records the fingerprint information, but also the virtual address and additional information of the data identified by the fingerprint information. Fig.11As shown, the memory can record a set of information for each data, and the set of information includes the fingerprint information, virtual address and additional information (tag) of the data. The additional information can be metadata related to the fingerprint information, such as persist information, stable information, dupl information and remove information. The persist information is used to indicate whether the group of information has been persisted. The stable information is used to indicate the unique key value (KV) of the data identified by the fingerprint information. Here, the key in the key value is the fingerprint information of the data, and the value in the key value is the address of the physical block used to store the data. The dupl information is used to indicate whether the data identified by the fingerprint information is duplicate data that has been accessed recently. The remove information is used to indicate whether the data indicated by the fingerprint information is valid data.

[0115] After obtaining the virtual address assigned to the data to be written, the virtual address can also be recorded in the memory. When the memory manages the fingerprint information according to the access popularity of the written data, after the data to be written is written to the physical address, the fingerprint information and virtual address of the data to be written can also be recorded at the head of the memory bucket. In addition, after the data to be written is written to the physical address, additional information of the data to be written can also be recorded according to the writing status of the data to be written to save metadata related to the fingerprint information. Optionally, the metadata related to deduplication in this application and the metadata of the file system are uniformly stored in the same device. In this way, during the reading and writing process of data, the metadata can be accessed from the device to obtain the relevant address information, and then the corresponding read and write operations can be performed in the device. There will be no problem in the related technology that the relevant operations need to be performed on two devices separately, and the performance of the file system can be guaranteed.

[0116] After the file system layer completes the relevant processing of the above step 803, it can submit the processed memory page to the IO request, such as submitting the IO request to the block layer IO (Block IO, bio) queue, so that the block layer continues to execute the subsequent writing process of the data to be written based on the allocated physical address.

[0117] Step 804: When the data to be written is duplicate data, the file system layer obtains the virtual address of the duplicate data to be written, and records the corresponding relationship between the virtual address and the logical address of the data to be written.

[0118] When the data to be written is duplicate data, there is no need to perform the operation of writing the data to be written into the physical block, but it is necessary to record relevant information to record the write event of the data to be written. At this time, the virtual address of the duplicate data can be obtained, and the correspondence between the virtual address and the logical address of the data to be written can be recorded. By recording the correspondence between the virtual address and the logical address of the data to be written, since the virtual address corresponds to the physical address, according to the correspondence, it can be known that the data recorded at the physical address corresponding to the virtual address is also the data of the logical address. Among them, the file system layer can query the correspondence between the fingerprint information and the virtual address based on the fingerprint information of the duplicate data to obtain the virtual address of the duplicate data. The correspondence between the fingerprint information and the virtual address is obtained during the writing process of the duplicate data.

[0119] In addition, in order to manage the consistency of the written data, the file system layer can also record or update the total number of logical addresses corresponding to the virtual address each time the data is written. For example, when the data to be written is duplicate data, since the total number of logical addresses corresponding to the virtual address of the duplicate data has been recorded during the writing process of the duplicate data, the total number of logical addresses corresponding to the virtual address can be updated based on the total number of logical addresses corresponding to the virtual address originally recorded. Here, updating the total number actually means adding one to the original total number.

[0120] According to the previous content, the memory not only records the fingerprint information, but also the virtual address of the data identified by the fingerprint information. Therefore, when the memory manages the fingerprint information according to the access popularity of the written data, after obtaining the virtual address of the duplicate data to be written, the fingerprint information and virtual address of the data to be written can be moved to the head of the memory bucket. In addition, additional information of the data to be written can also be recorded to save metadata related to the fingerprint information.

[0121] It should be noted that when the data to be written is duplicate data, although there is no need to execute the operation of writing the data to be written into the physical block, a dirty mark of the data to be written will still exist in the memory page. After the file system layer completes the relevant processing of the above step 804, it can clear the dirty mark of the memory page and terminate the write back operation to complete the writing process of the data to be written.

[0122] According to the execution process of step 804, when the data to be written is duplicate data, the file system can delete the duplicate data on the data writing path, thereby reducing the amount of written data and the storage space occupied, reducing useless writing, and helping to increase the life and storage capacity of the storage device.

[0123] The above describes the data writing process of the file system, and the following describes the data migration process. According to the previous description, the impact of virtual addresses on data management is mainly reflected in the data writing process and the data migration process. A typical application of data migration is to migrate valid data in the physical block to be recycled during the garbage collection process. Therefore, the following takes the application of data migration in the garbage collection process as an example to explain the implementation process of data migration.

[0124] Fig.12 This is a flowchart of a garbage collection process provided by an embodiment of the present application. Fig.12 As shown, the garbage collection process includes the following steps:

[0125] Step 1101: The file system layer obtains the first physical address of the data to be migrated in the external memory.

[0126] The data to be migrated is the data that needs to be migrated from the current physical block to other physical blocks. When data migration is applied in the garbage collection process, the data to be migrated is the valid data stored in the physical block to be reclaimed. For example, because F2FS organizes the storage area into a log structure, when the remaining free space in the storage area of ​​F2FS is insufficient, it is necessary to migrate the valid data in the old space and reclaim the old space. At this time, the data to be migrated is the valid data in the old space. In one implementation, the file system layer can determine the physical blocks to be reclaimed according to its garbage collection strategy, and determine the valid data in the physical blocks to be reclaimed. The valid data is the data to be migrated. After determining the data to be migrated, its first physical address can be determined. For example, the physical address of the physical block storing the data to be migrated is the first physical address. Fig.13 As shown, the physical block with physical address 5 is the physical block to be reclaimed, and the valid data stored in the physical block is data 2A. The data to be migrated is data 2A, and its first physical address is 5. Fig.13 The circled numbers indicate the order of operations.

[0127] Step 1102: The file system layer obtains the virtual address of the data to be migrated based on the first physical address.

[0128] After obtaining the first physical address of the data to be migrated, the virtual address of the data to be migrated can be obtained based on the first physical address. In one implementation, the correspondence between the physical address and the virtual address of the data to be migrated can be queried based on the first physical address to obtain the virtual address of the data to be migrated. For example, Fig.13 As shown, by querying the correspondence between the physical address and the virtual address of the data to be migrated, it is obtained that the virtual address of the data to be migrated is 800.

[0129] In one implementation, the file system layer may use SSA to maintain a mapping from a physical address to a virtual address. When executing step 1102, the SSA may be queried based on the first physical address to obtain a virtual address corresponding to the first physical address. In the present application, the SSA may be obtained by modifying the original SSA in F2FS. Fig.14 Schematic diagram of a metadata layout provided by an embodiment of the present application. Fig.14 It can be seen that the layout of SSA in the metadata has not changed.

[0130] Step 1103: The file system layer writes the data to be migrated to the second physical address.

[0131] After determining the data to be migrated, the file system layer can obtain the index of the data to be migrated, index the data block where the data to be migrated is located based on the index, read the data block into the memory based on the index result, and then write the data block from the memory to the second physical address to complete the migration process of the data to be migrated. The second physical address is the physical address reallocated for the data to be migrated. For example, Fig.13 As shown, the second physical address is 17.

[0132] Optionally, before writing the data to be migrated to the second physical address, the total number of logical addresses corresponding to the virtual address of the data to be migrated can be obtained, and based on the total number, it is determined whether to execute the process of writing the data to be migrated to the second physical address. If the total number is 0, it means that all logical references to the data to be migrated have become invalid during the migration, and the data to be migrated has become invalid data, and there is no need to migrate the data to be migrated. If the total number is not 0, it means that the data to be migrated is still valid data, and it is necessary to migrate the data to be migrated. Therefore, when the total number of logical addresses corresponding to the virtual address of the data to be migrated is not 0, the data to be migrated is written to the second physical address. When the total number of logical addresses corresponding to the virtual address of the data to be migrated is 0, the migration operation of the data to be migrated is stopped, and the data to be migrated can be discarded at this time. In this way, the amount of data migration can be reduced, which helps to shorten the time consumption of data migration and helps to reduce the time consumed by garbage collection. In order to manage data consistency, after the data to be migrated is read into the memory, the reference count of the data to be migrated can be increased by one, and before executing the operation of writing the data to be migrated to the second physical address, the reference count of the data to be migrated can be reduced by one, and then it is determined whether the reference count of the data to be migrated is 0.

[0133] Step 1104: The file system layer updates the correspondence between the virtual address and the physical address of the data to be migrated based on the second physical address.

[0134] After the file system layer writes the data to be migrated to the second physical address, the corresponding relationship between the virtual address and the physical address of the data to be migrated can be updated according to the second physical address. Fig.13 As shown, the physical address corresponding to the virtual address 800 is changed from 5 to 17.

[0135] In one implementation, the file system layer may use VAT to maintain a mapping from a virtual address to a physical address. When executing step 1104, the physical address corresponding to the virtual address may be changed to the second physical address in the VAT. In the present application, the VAT may be obtained by modifying the original NAT in the F2FS. Fig.14 Schematic diagram of a metadata layout provided by an embodiment of the present application. Fig.14 It can be seen that NAT is not included in the metadata, and VAT appears instead.

[0136] After completing the migration of the data to be migrated in the old space, the old space can be reclaimed. The data migration process can also be applied to the defragmentation process. By migrating the data in the storage space with lower data storage density, the storage space can be defragmented. However, in some cases, deduplication may conflict with defragmentation to a certain extent. First, in general, file defragmentation is achieved by reading the files out and then writing them back uniformly. In the deduplication file system, since deduplication is also performed when writing back, it may not actually have the effect of defragmentation. Second, even if defragmentation can be performed normally, after defragmentation for a certain file, the file restores the order, but if the logical blocks of other files are indexed to the physical blocks corresponding to this file, the order of these files will be destroyed. For this reason, the present application can support defragmentation by modifying the mapping of virtual addresses to physical addresses on the basis of realizing the deduplication file system. At the same time, if there is a significant difference in access heat between the logical blocks mapped to the same physical block, the order of files with higher access heat can be prioritized.

[0137] It can be seen that the file system layer allocates virtual addresses for data and records the correspondence between virtual addresses and physical addresses, as well as the correspondence between virtual addresses and logical addresses, so that virtual addresses can isolate logical addresses and physical addresses, avoiding direct mapping of logical addresses to physical addresses. In addition, since the virtual addresses of duplicate data are the same, virtual addresses and physical addresses can form a one-to-one mapping, which reduces the complexity of the file system mapping relationship, helps to improve the processing speed of garbage collection, and ensures the read and write performance of the file system.

[0138] When migrating data, what needs to be modified is the correspondence between the virtual address and the physical address, and there is no need to modify the correspondence between the logical address and the physical address as in the related art. In addition, since the virtual addresses of the duplicate data are the same, the virtual address and the physical address can form a one-to-one mapping. When modifying the correspondence between the virtual address and the physical address, no matter whether the data to be migrated is unique data or duplicate data, only the virtual address needs to be modified, and there is no need to modify multiple logical addresses corresponding to the duplicate data as in the related art, so the migration speed of the data can be guaranteed and the performance of the file system can be guaranteed. In addition, according to the above migration process, it can be known that in the migration process, no matter whether the data to be migrated is unique data or duplicate data, the data migration method provided by the embodiment of the present application only needs to perform one read process and one write process on the migrated data. In the related art, when the data to be migrated is unique data, it is necessary to perform multiple read processes and multiple write processes on the migrated data. When the data to be migrated is duplicate data, it is necessary to perform repeated operations on each data block, and the number of read processes and write processes required to be performed is greater. For example, in the garbage collection process of the native F2FS, at least three read processes and two write processes are required to migrate the unique data. Therefore, the data migration method provided in the embodiment of the present application can further ensure the data migration speed.

[0139] In addition, since the data management method provided by the present application is executed by the file system layer, it will perform a deduplication process on the data during the data writing process, and can perceive the deduplication semantics, so it will not store the problem of false storage space in the related technology. In addition, the present application unifies the deduplication index and data address mapping into the file mapping structure of the file system itself for management, avoiding the space overhead of additional storage of the deduplication index, and reducing the impact of the additional file operation process introduced by deduplication on the file system performance.

[0140] Furthermore, since the present application uses the fingerprint information of the data stored in the memory, compared with the fingerprint information of the data stored in the external memory, it can effectively reduce the impact of reading the fingerprint information on the read and write performance of the file system. Moreover, when the memory uses multiple memory sets to store the fingerprint information, when the fingerprint information of the information to be written is matched with the fingerprint information of the data already written in the memory, it can first perform a preliminary search in the multiple memory sets based on the fingerprint prefix of the fingerprint information, and then match the fingerprint information in the retrieved memory sets, which effectively reduces the total number of fingerprint information that needs to be matched, improves the speed of matching the fingerprint information, and helps to improve the performance of the file system.

[0141] It should be noted that the sequence of steps of the data management method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0142] The above describes the data management method of the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a data management device. The structural diagram of the data management device is shown in FIG. Fig.15 As shown. The device is applied to the file system layer. It should be understood that the system may include more additional nodes than the structure shown or omit some of the nodes shown therein, and the embodiment of the present application is not limited to this. Fig.15 As shown, the data management device 150 includes:

[0143] The acquisition module 1501 is used to acquire the data to be written in the file to be written into the external memory.

[0144] Processing module 1502 is used to allocate a virtual address and a physical address to the data to be written when the data to be written is not duplicate data, record the data to be written in the physical address, record the correspondence between the virtual address and the physical address, and the correspondence between the virtual address and the logical address of the data to be written, and the logical address is used to indicate the position of the data to be written in the file.

[0145] The processing module 1502 is further configured to obtain the virtual address of the duplicate data of the data to be written when the data to be written is duplicate data, and record the corresponding relationship between the virtual address and the logical address of the data to be written.

[0146] Optionally, the processing module 1502 is further used to: when the data to be written is not duplicate data, record the total number of logical addresses corresponding to the virtual address; when the data to be written is duplicate data, update the total number of logical addresses corresponding to the virtual address.

[0147] Optionally, the acquisition module 1501 is also used to acquire the first physical address of the data to be migrated in the external memory; the acquisition module 1501 is also used to acquire the virtual address of the data to be migrated based on the first physical address; the processing module 1502 is also used to write the data to be migrated to the second physical address; the processing module 1502 is also used to update the correspondence between the virtual address and the physical address of the data to be migrated based on the second physical address.

[0148] Optionally, the processing module 1502 is specifically configured to: when the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is not 0, write the data to be migrated into the second physical address.

[0149] Optionally, the processing module 1502 is further configured to: when the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is 0, stop the migration operation on the data to be migrated.

[0150] Optionally, the data to be migrated is valid data stored in the physical block to be reclaimed.

[0151] Optionally, the processing module 1502 is specifically configured to obtain a virtual address of duplicate data of the data to be written based on a correspondence between the data to be written and the virtual address.

[0152] Optionally, the processing module 1502 is also used to: obtain fingerprint information of the data to be written; match the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory; when the fingerprint information of the data to be written matches the fingerprint information of the written data, determine that the data to be written is duplicate data; when the fingerprint information of the data to be written does not match the fingerprint information of the written data, determine that the data to be written is not duplicate data.

[0153] Optionally, the memory stores fingerprint information of the written data in descending order of access popularity of the written data.

[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each node described above can refer to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0155] The embodiment of the present application provides a computing device, which is used to implement part or all of the functions of the data management method provided in the embodiment of the present application. Fig.16 Schematic diagram of a computing device provided in an embodiment of the present application. Fig.16 As shown, the computing device 1600 includes a processor 1601, a memory 1602, a communication interface 1603 and a bus 1604. The processor 1601, the memory 1602 and the communication interface 1603 are connected to each other through the bus 1604.

[0156] Processor 1601 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. A dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a network processor (NP). Processor 1601 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the data management method of the present application may be completed by the hardware integrated logic circuit in processor 1601 or instructions in software form.

[0157] The memory 1602 is used to store computer programs, which include an operating system 1602a and executable code (i.e., program instructions) 1602b. The memory 1602 is, for example, a read-only memory or other types of static storage devices that can store static information and instructions, or a random access memory or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a blue-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory 1602 is used to store an outbound port queue, etc. The memory 1602 is, for example, independent and connected to the processor 1601 via a bus 1604. Or the memory 1602 and the processor 1601 are integrated together. The memory 1602 may store executable code. When the executable code stored in the memory 1602 is executed by the processor 1601, the processor 1601 is used to perform some or all functions of the data management method provided in the embodiment of the present application. For the implementation of the process executed by the processor 1601, please refer to the relevant description in the aforementioned embodiment. The memory 1602 may also include software modules and data required for other running processes such as an operating system.

[0158] The communication interface 1603 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 1603 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0159] The bus 1604 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the memory 1602, the processor 1601, and the communication interface 1603). For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device through the bus 1604 as an example. Optionally, the above-mentioned devices inside the computing device 1600 can also be connected to each other in communication with each other using other connection methods other than the bus 1604. For example, the above-mentioned devices inside the computing device 1600 are interconnected through an internal logical interface.

[0160] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0161] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing the program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the data management method provided in the embodiments of the present application are implemented.

[0162] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0163] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0164] Optionally, the structure of at least one computing device included in the computing device cluster can be found in Fig.16 The computing device 1600 is shown. The memory 1602 in one or more computing devices 1600 in the computing device cluster may store the same instructions for executing the data management method.

[0165] In some possible implementations, the memory 1602 of one or more computing devices 1600 in the computing device cluster may also store some instructions for executing the data management method. In other words, the combination of one or more computing devices 1600 may jointly execute the instructions for executing the data management method.

[0166] It should be noted that the memory 1602 in different computing devices 1600 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data management device. That is, the instructions stored in the memory 1602 in different computing devices 1600 can implement the functions of one or more modules in the above modules.

[0167] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.17 A possible implementation is shown. Fig.17 As shown, two computing devices 1700A and 1700B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, computing devices 1700A and 1700B include a bus 1702, a processor 1704, a memory 1706, and a communication interface 1708. For example, computing device 1700A can perform the function of the acquisition module in the embodiment of the present application, and computing device 1700B can perform the function of the processing module in the embodiment of the present application.

[0168] It should be understood that Fig.17 The functions of the computing device 1700A shown in the figure can also be completed by multiple computing devices 1700. Similarly, the functions of the computing device 1700B can also be completed by multiple computing devices 1700. And the deployment mode of the modules used to implement the data management method in the computing device can also be adjusted according to the application requirements.

[0169] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements the data management method provided in the embodiment of the present application.

[0170] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the data management method provided by the embodiment of the present application.

[0171] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0172] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and executable code involved in this application are all obtained with full authorization.

[0173] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means a plurality, unless otherwise expressly defined.

[0174] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0175] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0176] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0177] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data management method, characterized in that: The method is applied to the file system layer, and the method comprises: Obtaining data to be written in a file to be written into an external storage device; When the data to be written is not duplicate data, a virtual address and a physical address are allocated to the data to be written, the data to be written is recorded in the physical address, a corresponding relationship between the virtual address and the physical address, and a corresponding relationship between the virtual address and a logical address of the data to be written is recorded, and the logical address is used to indicate a position of the data to be written in the file; When the data to be written is duplicate data, a virtual address of the duplicate data to be written is obtained, and a corresponding relationship between the virtual address and the logical address of the data to be written is recorded.

2. The method according to claim 1, characterized in that The method further comprises: When the data to be written is not duplicate data, recording the total number of logical addresses corresponding to the virtual address; When the data to be written is duplicate data, the total number of logical addresses corresponding to the virtual address is updated.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Acquire a first physical address of the data to be migrated in the external memory; Based on the first physical address, obtaining a virtual address of the data to be migrated; Writing the data to be migrated into a second physical address; Based on the second physical address, the correspondence between the virtual address and the physical address of the data to be migrated is updated.

4. The method according to claim 3, characterized in that The step of writing the data to be migrated into the second physical address comprises: When the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is not 0, the data to be migrated is written into the second physical address.

5. The method according to claim 3, characterized in that After acquiring the virtual address of the data to be migrated based on the first physical address, the method further includes: When the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is 0, the migration operation on the data to be migrated is stopped.

6. The method according to any one of claims 1 to 5, characterized in that: The data to be migrated is valid data stored in the physical block to be reclaimed.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining the virtual address of the duplicate data of the data to be written comprises: Based on the correspondence between the query data of the data to be written and the virtual address, the virtual address of the duplicate data of the data to be written is obtained.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining fingerprint information of the data to be written; Matching the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory; When the fingerprint information of the data to be written matches the fingerprint information of the written data, determining that the data to be written is duplicate data; When the fingerprint information of the data to be written does not match the fingerprint information of the written data, it is determined that the data to be written is not duplicate data.

9. The method according to claim 8, characterized in that The memory stores the fingerprint information of the written data in descending order of access heat of the written data.

10. A data management device, characterized in that: The device is applied to the file system layer, and the device includes: An acquisition module, used for acquiring data to be written in a file to be written into an external storage device; a processing module, configured to allocate a virtual address and a physical address to the data to be written when the data to be written is not duplicate data, record the data to be written in the physical address, record the corresponding relationship between the virtual address and the physical address, and the corresponding relationship between the virtual address and the logical address of the data to be written, wherein the logical address is used to indicate the position of the data to be written in the file; The processing module is further configured to obtain a virtual address of the duplicate data of the data to be written when the data to be written is duplicate data, and record a corresponding relationship between the virtual address and the logical address of the data to be written.

11. The device according to claim 10, characterized in that The processing module is further used for: When the data to be written is not duplicate data, recording the total number of logical addresses corresponding to the virtual address; When the data to be written is duplicate data, the total number of logical addresses corresponding to the virtual address is updated.

12. The device according to claim 10 or 11, characterized in that The acquisition module is further used to acquire a first physical address of the data to be migrated in the external memory; The acquisition module is further configured to acquire a virtual address of the data to be migrated based on the first physical address; The processing module is further used to write the data to be migrated into a second physical address; The processing module is further configured to update the correspondence between the virtual address and the physical address of the data to be migrated based on the second physical address.

13. The device according to claim 12, characterized in that The processing module is specifically used for: When the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is not 0, the data to be migrated is written into the second physical address.

14. The device according to claim 12, characterized in that The processing module is further used for: When the total number of logical addresses corresponding to the virtual addresses of the data to be migrated is 0, the migration operation on the data to be migrated is stopped.

15. The device according to any one of claims 10 to 14, characterized in that The data to be migrated is valid data stored in the physical block to be reclaimed.

16. The device according to any one of claims 10 to 15, characterized in that: The processing module is specifically used for: Based on the correspondence between the query data of the data to be written and the virtual address, the virtual address of the duplicate data of the data to be written is obtained.

17. The device according to any one of claims 10 to 16, characterized in that: The processing module is further used for: Obtaining fingerprint information of the data to be written; Matching the fingerprint information of the data to be written with the fingerprint information of the written data stored in the memory; When the fingerprint information of the data to be written matches the fingerprint information of the written data, determining that the data to be written is duplicate data; When the fingerprint information of the data to be written does not match the fingerprint information of the written data, it is determined that the data to be written is not duplicate data.

18. The device according to claim 17, characterized in that The memory stores the fingerprint information of the written data in descending order of access heat of the written data.

19. A computing device cluster, characterized in that: The method comprises at least one computing device, each computing device comprises a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images