Method and electronic device for data deduplication

Abstract

Embodiments of the present application provide a data deduplication method, which is applied to an electronic device and includes: obtaining, by the electronic device, first information of an application; the first information including a write I / O size and a written page offset; obtaining, by the electronic device, a first memory data block according to the first information; calculating, by the electronic device, a first data fingerprint of the first memory data block; determining, by the electronic device, whether there is a corresponding first physical address of the first data fingerprint; and when it is determined that there is the corresponding first physical address of the first data fingerprint, updating, by the electronic device, a mapping relationship between the first memory data block and the first physical address. This technical solution can achieve zero-copy overhead or close-to-zero-copy overhead of repeated data when the electronic device uses a file system for data deduplication.

Description

Technical Field

[0001] This application relates to the field of data deduplication technology, and more specifically, to a data deduplication method and electronic device. Background Technology

[0002] In the field of data deduplication, users can use flash controllers and file systems for deduplication. When using flash controllers, the duplicate block lookup process can directly control the address mapping table to view blocks with the same data fingerprint, resulting in good deduplication performance and ease of implementation. However, this approach relies on specific hardware and is not flexible enough. When using file systems for deduplication, the file system's write input / output (I / O) transfer from the application to the kernel requires a data copy, meaning data is copied from user space to kernel space. Therefore, there is additional copy overhead for duplicate data. Summary of the Invention

[0003] This application provides a method and electronic device for data deduplication. This technical solution can achieve zero or near-zero copy overhead for duplicate data when using a file system for data deduplication.

[0004] Firstly, a data deduplication method is provided, applied to an electronic device. The method includes: the electronic device acquiring first information of an application; the first information including write I / O size and write page offset; the electronic device acquiring a first memory data block based on the first information; the electronic device calculating a first data fingerprint of the first memory data block; the electronic device determining whether a first physical address corresponding to the first data fingerprint exists; and when it is determined that a first physical address corresponding to the first data fingerprint exists, the electronic device updating the mapping relationship between the first memory data block and the first physical address. In this embodiment, the electronic device can acquire the first information of an application and acquire a first memory data block based on the first information. The first memory data block can be multiple memory data blocks. It calculates the data fingerprint of the first memory data block. When it is determined that a physical address corresponding to the data fingerprint exists, it can be determined that the data in the memory data block already exists. Therefore, the mapping relationship between the first memory data block and the first physical address can be directly updated without transmitting the data itself. This allows the technical solution to achieve zero or near-zero copy overhead for deduplication, thereby improving the speed of the application's write process.

[0005] In conjunction with the first aspect, in one implementation of the first aspect, the electronic device obtains a first memory data block based on the first information, including: the electronic device dividing the write I / O size according to the page size and the written page offset to obtain the first memory data block. In this embodiment, when the write I / O size generated by the application is large, the electronic device can divide the write I / O size according to the page size and the written page offset to obtain the first memory data block, which facilitates the electronic device performing subsequent data deduplication operations in units of data blocks.

[0006] For example, the page size is typically 4KB. A memory data block with a page size of 4KB is called a page-aligned data block.

[0007] It should be understood that when the file I / O size is less than or equal to the page size (e.g., 4KB), it is not necessary to split the file I / O; instead, the file I / O size can be treated as a single block of memory data.

[0008] In conjunction with the first aspect, in one implementation of the first aspect, when the first memory data block comprises multiple page-aligned memory data blocks, the electronic device calculates the first data fingerprint of the first memory data block, comprising: the electronic device performing a first calculation on the first memory data block in the user space of the application to obtain the first data fingerprint. In one embodiment of this application, when the first memory data block comprises page-aligned memory data blocks, the electronic device can directly calculate the corresponding first data fingerprint of the first memory data block in the user space of the application. This facilitates the electronic device in querying the corresponding physical address based on the first data fingerprint, without needing to transfer the actual data to the kernel space, thus achieving zero data copying.

[0009] In conjunction with the first aspect, in one implementation of the first aspect, when the first memory data block includes multiple page-aligned memory data blocks and one non-page-aligned memory data block, before the electronic device calculates the first data fingerprint of the first memory data block, the method further includes: the electronic device transferring data in the non-page-aligned memory data block from the user space of the application to the kernel space of the electronic device; and the electronic device converting the non-page-aligned memory data block into a page-aligned memory data block in the kernel space. In one embodiment of this application, when the first memory data block includes page-aligned memory data blocks and non-page-aligned memory data blocks, the electronic device can convert the non-page-aligned memory data block into a page-aligned memory data block in the kernel space. Therefore, the electronic device only needs to transfer data from one non-page-aligned memory data block from user space to kernel space, greatly reducing the amount of data transferred and essentially achieving zero data copying.

[0010] In conjunction with the first aspect, in one implementation of the first aspect, the first information further includes a file descriptor; before the electronic device updates the mapping relationship between the first memory data block and the first physical address, the method further includes: the electronic device transferring the file descriptor, the written page offset, and the first physical address from the user space of the application to the kernel space of the electronic device. In this embodiment, the electronic device transfers the file descriptor, the written page offset, and the first physical address from the user space to the kernel space, thereby eliminating the need to copy the data of the first memory data block itself to the kernel space, reducing data copying.

[0011] In conjunction with the first aspect, in one implementation of the first aspect, the electronic device updates the mapping relationship between the first memory data block and the first physical address by: updating the mapping relationship between the first memory data block and the first physical address in the kernel space according to the file descriptor, the written page offset, and the first physical address. In this embodiment, the electronic device can update the mapping relationship between the first memory data block and the first physical address according to the file descriptor, the written page offset, and the first physical address, thereby eliminating the need to copy the data of the first memory data block itself to the kernel space, reducing data copying, and improving the speed of the write process.

[0012] In conjunction with the first aspect, in one implementation of the first aspect, the electronic device stores a first mapping table, which is a mapping table between data fingerprints and physical addresses; determining the existence of a first physical address corresponding to the first data fingerprint includes: querying the first physical address from the first mapping table based on the first data fingerprint. In this embodiment of the application, the electronic device stores a mapping table between data fingerprints and physical addresses, thereby allowing convenient querying of the first physical address corresponding to the first data fingerprint from the mapping table, thus determining that the data in the data block corresponding to the first data fingerprint is duplicated.

[0013] In conjunction with the first aspect, in one implementation of the first aspect, the method further includes: when it is determined that there is no first physical address corresponding to the first data fingerprint, the electronic device allocates a first physical data block for the first memory data block; the electronic device writes the data in the first memory data block into the first physical data block. In this embodiment of the application, when it is determined that there is no first physical address corresponding to the first data fingerprint, it can be determined that the data in the first memory data block is not duplicated. At this time, the electronic device can allocate a new physical data block for storing its data.

[0014] In conjunction with the first aspect, in one implementation of the first aspect, the physical address corresponding to the first physical data block is the second physical address. The method further includes: the electronic device storing the mapping relationship between the first memory data block and the second physical address. Based on the embodiments of this application, the electronic device can store the mapping relationship between the first memory data block and the second physical address where its data is actually stored, thereby determining the actual storage location of the first memory data block.

[0015] In conjunction with the first aspect, in one implementation of the first aspect, the method further includes: the electronic device calculating a second data fingerprint of the first physical data block; and the electronic device saving the second data fingerprint and the second physical address to the first mapping table. Based on the embodiments of this application, the electronic device saves the second data fingerprint and the second physical address to the first mapping table to update the first mapping table, thereby ensuring accuracy in the next data deduplication process.

[0016] In conjunction with the first aspect, in one implementation of the first aspect, the method further includes: the electronic device mapping the first mapping table from the kernel space of the electronic device to the user space of the application. In one embodiment of this application, the electronic device maps the first mapping table from the kernel space to the user space of the application, thereby ensuring that the first mapping table is shared in user mode and kernel mode, thereby increasing the speed of searching for the physical address corresponding to a data block in user mode.

[0017] In a second aspect, an electronic device is provided, including modules for implementing a data deduplication method as described in the first aspect and any possible implementation thereof.

[0018] Thirdly, an electronic device is provided, comprising: one or more processors; one or more memories; said one or more memories storing one or more computer programs, said one or more computer programs including instructions that, when executed by said instructions by said processors, cause a data deduplication method as described in the first aspect and any possible implementation thereof to be performed.

[0019] Fourthly, a chip is provided, the chip including a processor and a communication interface, the communication interface being used to receive signals and transmit the signals to the processor, the processor processing the signals such that a data deduplication method as described in the first aspect and any possible implementation thereof is executed.

[0020] Fifthly, a computer-readable storage medium is provided, wherein computer instructions are stored therein, which, when executed on a computer, cause a data deduplication method as described in the first aspect and any possible implementation thereof to be performed.

[0021] In a sixth aspect, a computer program product is provided, including computer instructions that, when executed on a computer, cause a data deduplication method as described in the first aspect and any possible implementation thereof to be performed. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a system architecture to which the embodiments of this application may be applied.

[0023] Figure 2 This is a schematic flowchart of a data deduplication method provided in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of an address mapping relationship provided in an embodiment of this application.

[0025] Figure 4 This is a schematic flowchart of a data deduplication method provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] Before introducing the technical solution of this application, a brief introduction will be given to some of the technical terms involved in the embodiments of this application.

[0028] User space: Provides the space where applications run. When a process runs in user space, it is said that the process is in user running mode (or simply user mode).

[0029] Kernel space: The location where kernel code runs. Processes running in this space have more access privileges to the system than processes running in user space. When a task (process) executes a system call and gets trapped in kernel code, the process is said to be in kernel running mode (or simply kernel mode).

[0030] Logical address: In computer architecture, this refers to the address of a memory cell, storage element, or network host as seen from the application's perspective.

[0031] Physical address: The address where data is loaded into memory; the actual address of a memory cell. For example, the address where the data block actually resides.

[0032] Page alignment: When memory performs paging, it typically finds the boundary of each 4KB address and performs paging operations. A page is generally 4KB in size. In this embodiment, write I / O generated by the application is divided into multiple data blocks of 4KB each. I / O blocks of 4KB each can be page-aligned. Conversely, I / O blocks smaller than 4KB after partitioning can be non-page-aligned.

[0033] Inode: Each file corresponds to one inode, and each inode has a number. The operating system uses the inode number to identify different files.

[0034] Page offset: If each page size is 2 k B, if the logical address is identified by a binary number, then the last k bits are the page offset, and the rest are the page number.

[0035] File descriptor (FD): In the system, most I / O operations are assigned an integer as a number, which can be used as the file descriptor for that I / O operation.

[0036] Portable operating system interface of UNIX (POSIX): An interface used for porting between different operating systems.

[0037] In the field of data deduplication, users can use flash controllers and file systems for deduplication. When using flash controllers, the duplicate block lookup process can directly control the address mapping table to view blocks with the same data fingerprint, resulting in good deduplication performance and ease of implementation. However, this approach is hardware-dependent and lacks flexibility. When using file systems for deduplication, the file system's write I / O from the application to the kernel requires a data copy, i.e., copying data from user space to kernel space. Therefore, there is additional copying overhead for duplicate data.

[0038] In view of this, embodiments of this application provide a method and electronic device for data deduplication, which can achieve zero copy overhead for duplicate data when using a file system for data deduplication.

[0039] Figure 1 This is a schematic diagram of a system architecture to which embodiments of this application may be applied. For example... Figure 1 As shown, this data deduplication method can be applied to an electronic device 100, which may include multiple installed applications 110, a user-mode deduplication module 120, and a kernel space 130. The kernel space 130 may include a page cache layer 131 and a kernel deduplication file system 150. The applications may be, for example, Huawei Music, Huawei Video, Huawei App Market, etc., but this embodiment does not limit the specific applications.

[0040] The kernel deduplication file system 150 may include a deduplication module B132, an FP-PA mapping table 140a, a calculation module 136, an allocation module 135, a write module 134, and an update module 133. In some embodiments, the deduplication file system 150 may also include some of the above modules, which is not limited in this application.

[0041] It is understood that the user-space deduplication module 120 can access the FP-PA mapping table 140a in a read-only manner. For example, the electronic device maps the FP-PA mapping table 140a to the user-space deduplication module 120 through direct mapping to obtain a read-only FP-PA mapping table 140b.

[0042] In some embodiments, when an application 110 needs to write data, the electronic device 100 can determine whether the data to be written is duplicate data. If the data is duplicate data, the physical address corresponding to the duplicate data can be updated directly, thereby eliminating the need to pass the duplicate data itself, reducing data copying, and thus speeding up the write process.

[0043] In some embodiments, when application 110 generates a write I / O, the I / O segmentation module 121 in the user-space deduplication module 120 can segment the size of the write I / O to obtain multiple page-aligned data blocks (e.g., 4KB in size). The deduplication module A122 can calculate the data fingerprint of each data block according to a data fingerprint algorithm, and look up the corresponding physical address from the fingerprint-physical address (FP-PA) mapping table 140b based on the data fingerprint. Once the physical address corresponding to the data block is found in the FP-PA mapping table 140b, it means that the data in that data block already exists. The deduplication module A122 can then directly transmit the data fingerprint, physical address, page offset, file descriptor, and other information of that data block to the kernel deduplication file system 150 via the system interface, instead of transmitting the actual data of the data block. The update module 133 in the kernel deduplication file system 150 updates the physical address of the file's inode based on the received data fingerprint, physical address, page offset, file descriptor, and other information.

[0044] Based on the technical solutions provided in the above embodiments, the application can determine in user space that the data corresponding to the application's write I / O is duplicated, thus eliminating the need to copy the specific data of the data block from user space to kernel space, thereby avoiding additional data copying and accelerating file deduplication and read / write speeds.

[0045] For example, the inode of a file can be a hash map or hash table, where the key in the inode is the page offset and the value is the physical address where the file data is located. If an application needs to write 4096 bytes of data from the 4097th byte of a file (the page offset), then the page offset can be calculated by dividing 4097 by the page size of 4096 bytes (4KB), resulting in a page offset of 1 with a remainder of 1. This means writing 4096 bytes of data starting from the first byte of the second page of the file. Therefore, this second page is page-aligned. The data fingerprint A of this second page is then calculated. When this data fingerprint A has a corresponding physical address A, the update module 133 can obtain the inode of the file based on the file descriptor FD and update the mapping relationship (key-value) between the page offset 1 and the physical address A into the inode. This allows for zero-copy deduplication without transmitting the actual data, thereby improving the speed of data reading and writing.

[0046] It should be understood that the data in a page can represent a data block.

[0047] It should be understood that the inode of the file can also be an array, skip list, B-tree, or other data structure, and this application embodiment does not limit this.

[0048] In some embodiments, the user-space deduplication module 120 can intercept POSIX functions of the application 110 using the LD-PRELOAD identifier to obtain the write I / O size to be written. The POSIX function can be an open function, write function, pwrite function, lseek function, or close function, etc. It should be understood that the application is unaware of obtaining the application's write I / O size using LD-PRELOAD.

[0049] The deduplication module A122 passes the data fingerprint, physical address, page offset, file descriptor, and other information of the data block to the kernel deduplication file system 150. The system interface used can be an input and output control (ioctl) interface, which can avoid the overhead of a long I / O stack.

[0050] In some embodiments, after the I / O segmentation module 121 prunes the size of a write I / O, it may obtain multiple page-aligned data blocks and one non-page-aligned data block B. For example, if the size of a write I / O is 13KB, after pruning the size of the write I / O, it can obtain three page-aligned data blocks (e.g., 4KB in size) and one non-page-aligned data block B (e.g., 1KB in size). For the non-page-aligned data block B, the I / O segmentation module 121 transmits it to the kernel space 130 through a system interface (e.g., POSIX), where the page cache layer 131 in the kernel space 130 converts the non-page-aligned data block B into a page-aligned data block B.

[0051] In some embodiments, the page cache layer 131 may pad the data block B with zeros to bring its size to 4KB. For example, it may transform the data block B into a page-aligned data block.

[0052] In some embodiments, the deduplication module B132 in the kernel deduplication file system can calculate the data fingerprint B of data block B according to the data fingerprint algorithm, and query the physical address corresponding to the data fingerprint B from the FP-PA mapping table 140a. Once the physical address B corresponding to data block B is found in the FP-PA mapping table 140a, it means that the data in data block B already exists. The deduplication module B132 can then transmit the data fingerprint B, physical address B, written page offset, file descriptor, and other information to the update module 133. Correspondingly, the update module 133 can update the physical address of the file's inode based on the received data fingerprint B, physical address B, written page offset, file descriptor, and other information.

[0053] For non-page-aligned data blocks of a file, the actual data of the data block needs to be transferred to the kernel space for deduplication, which can ensure deduplication efficiency. Furthermore, the data size of the data block is relatively small (e.g., less than 4KB), thus having a smaller impact on file read and write efficiency.

[0054] For example, the inode of a file can be a hash map or a hash table, where the key in the inode is the page offset and the value is the physical address where the file data is located. Suppose an application needs to write 100 bytes of data from the 5000th byte of a file (the page offset). The page offset can be calculated by dividing 5000 by the page size of 4096 bytes (4KB), resulting in a page offset of 1 (corresponding to the second page), with a remainder of 904. This means writing 100 bytes of data starting from the 904th byte of the second page. Since pages are 4KB aligned, even if only 100 bytes of the second page are modified, the entire page needs to be rewritten, and then the data fingerprint B of that second page needs to be calculated. When the data fingerprint B has a corresponding physical address B, the update module 133 can obtain the inode of the file based on the file descriptor FD and update the mapping relationship (key-value) between page offset 1 and physical address B into that inode.

[0055] Understandably, the handling of page-aligned data blocks can be found in the description above.

[0056] It should be understood that the data fingerprint algorithm can be a secure hash algorithm, such as sha256, sha224, sha238, sha512, etc.

[0057] In some embodiments, when the deduplication module A122 does not find the corresponding physical address in the FP-PA mapping table 140b based on the data fingerprint, and / or when the deduplication module B132 does not find the corresponding physical address in the FP-PA mapping table 140a based on the data fingerprint B, it means that the data in the data block corresponding to the corresponding data fingerprint is not duplicated.

[0058] In one possible implementation, the I / O segmentation module 121 trims the size of write I / O generated by the application, resulting in page-aligned data blocks for all data blocks. When the deduplication module A122 fails to find a corresponding physical address in the FP-PA mapping table 140b based on the data block's data fingerprint, it can transfer the specific data block data to the allocation module 135 via a system interface (such as the ioctl interface). The allocation module 135 allocates a new data block for the data, and the write module 134 writes the data into the new data block. The update module 133 updates the physical address of the inode of the updated file. Furthermore, the calculation module 136 calculates the data fingerprint of the data block according to the data fingerprint algorithm and inserts the correspondence between the data fingerprint and the physical address into the FP-PA mapping table 140a.

[0059] In another possible implementation, the I / O segmentation module 121 trims the size of the write I / O generated by the application, resulting in several page-aligned data blocks and one non-page-aligned data block B. When the deduplication module B132 fails to find a corresponding physical address in the FP-PA mapping table 140b based on the data fingerprint of the data block, the allocation module 135 allocates a new data block for the specific data of data block B, and the write module 134 writes the specific data of data block B into the new data block. The update module 133 updates the physical address of the inode of this file. Furthermore, the calculation module 136 calculates the data fingerprint of the data block according to the data fingerprint algorithm and inserts the mapping relationship between the data fingerprint and the physical address into the FP-PA mapping table 140a.

[0060] Understandably, the FP-PA mapping table 140a is writable in the kernel deduplication file system 150. When the file data is unique, the computation module can calculate the data fingerprint of the data block and insert the mapping relationship between the data fingerprint and the physical address into the FP-PA mapping table 140a to update the FP-PA mapping table 140a. The FP-PA mapping table 140b, however, is read-only in the user-space deduplication module 120; that is, the deduplication module A is allowed to query it, but not to modify or edit it.

[0061] In some embodiments, the FP-PA mapping table 140b can be obtained by directly mapping the FP-PA mapping table 140a. For example, the memory mapping function mmap can be used to map the FP-PA mapping table 140a to the user-space deduplication module 120. In this case, the FP-PA mapping table is shared in user space and kernel space, thereby increasing the speed of searching for the physical address corresponding to a data block in user space.

[0062] It should be understood that the FP-PA mapping table 140a may be constructed in non-volatile memory (NVM), dynamic random access memory (DRAM), or flash memory, and this application embodiment does not limit it.

[0063] In some embodiments, the FP-PA mapping table 140b can be a hash table of any data structure, such as a table, a B-tree, etc.

[0064] It should be understood that since each application has a private user process address space, multiple applications can share a kernel space. In some embodiments, the FP-PA mapping table 140b can be set to read-only mode to prevent one application from modifying the FP-PA mapping table in user space and thus affecting other applications, ensuring that the FP-PA mapping table 140a in kernel space and the FP-PA mapping table 140b in user space are the same.

[0065] It is understood that the allocation module 135, the write module 134, and the calculation module 136 can also be the same module, which can implement the corresponding functions of the allocation module 135, the write module 134, and the calculation module 136. This application embodiment does not limit this.

[0066] Figure 2 This is a schematic flowchart illustrating a method for deduplicating files according to an embodiment of this application. This method can be applied to electronic devices, such as... Figure 2 As shown, the method 200 may include, but is not limited to, steps 301 to 310 as illustrated below.

[0067] 301, Get the write I / O size and the page offset written by the first application.

[0068] For example, the electronic device intercepts the POSIX function of the first application using the LD_PRELOAD flag to obtain the write I / O size and page offset generated by the first application. For instance, the write I / O size may also include the specific data buffer to be written in bytes. The user-space deduplication module in the electronic device obtains the write I / O size of the first application as 13312 bytes (13KB) and the page offset as 4097 bytes using the LD_PRELOAD flag. It should be understood that this 13312 bytes of data can actually be stored in the data buffer.

[0069] 302, divide the write I / O size to obtain the first memory data block.

[0070] In one example, when the acquired write I / O size is large, the electronic device can divide the write I / O size into segments, such that the resulting first memory data blocks are aligned with kernel pages. Each memory data block in this first memory data block is typically 4KB in size. For example, if the write I / O size is 12KB, the electronic device can divide the write I / O size into three 4KB memory data blocks.

[0071] In another example, when the acquired write I / O size is large, the electronic device can divide the acquired write I / O size into multiple page-aligned memory data blocks and one non-page-aligned memory data block. For example, if the write I / O size is 13KB, the electronic device can divide the write I / O size into three 4KB memory data blocks and one 1KB memory data block.

[0072] It should be understood that electronic devices divide the write I / O size by dividing the write I / O data buffer according to the write I / O size and the page offset to be written.

[0073] In other embodiments, when the write I / O size is less than or equal to the size of the memory data block aligned with the kernel page, it may not be necessary to segment the write I / O size. For example, if the size of the memory data block aligned with the kernel page is 4KB, then when the write I / O size is less than or equal to 4KB, it is not necessary to segment the write I / O size.

[0074] In other embodiments, the size of the memory data block aligned with the kernel page can also be other values, such as 8KB or 16KB.

[0075] It should be understood that the first memory data block is the memory data block obtained after dividing the write I / O size.

[0076] 303, Calculate the first data fingerprint of the first memory data block.

[0077] In some embodiments, the electronic device may calculate a first data fingerprint of a first memory data block according to a data fingerprinting algorithm. For example, the electronic device may calculate the first data fingerprint of the first memory data block according to a secure hashing algorithm, such as SHA256, SHA224, SHA238, SHA512, etc.

[0078] It should be understood that for memory data blocks aligned with kernel pages, their data fingerprints can be directly calculated. For memory data blocks not aligned with kernel pages, the data in the non-page-aligned memory data blocks can be padded with zeros to align with kernel pages, and then the data fingerprint of the memory data block can be calculated.

[0079] For example, a page cache layer located in the kernel space of an electronic device can pad non-page-aligned memory data blocks with zeros to make them page-aligned.

[0080] 304, Determine if the first physical address corresponding to the first data fingerprint exists in the FP-PA mapping table.

[0081] It should be understood that this FP-PA mapping table can be referred to in the previous description. For example, this FP-PA mapping table can be a hash table, where the key is the data fingerprint of the memory data block and the value is the physical address of the memory data block, so that the electronic device can look up the corresponding physical address through the data fingerprint.

[0082] In one example, the electronic device can query the FP-PA mapping table based on the first data fingerprint. When the query finds the first physical address corresponding to the first data fingerprint, it can be determined that the first physical address corresponding to the first data fingerprint exists in the FP-PA mapping table. Therefore, it can be determined that the data in the first memory data block corresponding to the first data fingerprint has already been stored, and the data is duplicated. The electronic device can then proceed to step 305.

[0083] In another example, the electronic device can query the FP-PA mapping table based on the first data fingerprint. If no result is found, it can be determined that the first physical address corresponding to the first data fingerprint does not exist in the FP-PA mapping table. It should be understood that the absence of the first data fingerprint and its corresponding first physical address in the FP-PA mapping table means that the data in the first memory data block corresponding to the first data fingerprint is unique, and the electronic device can then proceed to step 306.

[0084] 305, Update the physical address corresponding to the memory data block.

[0085] In some embodiments, after determining that a physical address corresponding to a data fingerprint exists in the FP-PA mapping table, the electronic device can directly update the physical address corresponding to the memory data block.

[0086] For example, the logical address of memory data block 1 is L1, and the physical address corresponding to its data fingerprint is P1. When the electronic device finds that the physical address corresponding to the data fingerprint of memory data block 2 is also P1, it means that memory data block 2 is a duplicate of memory data block 1. If the logical address of memory data block 2 is L2, then the physical address corresponding to L2 is updated to P1. Thus, the electronic device can avoid copying the actual data of the duplicate memory data blocks from user space to kernel space, avoiding additional data copying.

[0087] In some embodiments, each file has an inode that may include the physical addresses of the memory data blocks included in the file. For example, the inode of a file may include a logical address to physics address (L2P) mapping table for the file, so that updating the physical addresses of data blocks can be achieved by modifying the L2P mapping table.

[0088] Thus, in one embodiment of this application, the electronic device can acquire and segment the write I / O size generated by the application to obtain multiple memory data blocks, calculate the data fingerprint of each memory data block, and then query the corresponding physical address based on the data fingerprint. When the corresponding physical address is found, the physical address of the memory data block corresponding to the data fingerprint can be updated directly, thereby avoiding the need to copy the specific data of duplicate memory data blocks, avoiding additional copying overhead, and thus improving file read and write speed.

[0089] Understandably, when the electronic device determines that the write I / O generated by the first application is not duplicate data, the method 200 may further include steps 306 to 310.

[0090] 306, Allocate the first physical data block for the data in the first memory data block.

[0091] After determining that there is no first physical address corresponding to the first data fingerprint in the FP-PA mapping table, the electronic device can allocate a new first physical data block for the first memory data block. The first physical data block is used to store the data in the first memory data block, and the physical address of the first physical data block is the second physical address.

[0092] It should be understood that the physical address of the first physical data block is the location on the disk where the data in the first memory data block is actually stored.

[0093] 307, Write the data in the first memory data block into the first physical data block.

[0094] The electronic device writes the data from the first memory data block into the first physical data block. Then, it executes step 305 to update the physical address of the first physical data block. For example, if the logical address of the first memory data block is L3, and its data is stored in the first physical data block with a corresponding physical address of P3, then the physical address corresponding to L3 is updated to P3.

[0095] 308, Calculate the second data fingerprint of the first physical data block.

[0096] In some embodiments, the electronic device may calculate a second data fingerprint of the first physical data block according to a data fingerprint algorithm. For example, it may calculate a second data fingerprint of the first physical data block according to a secure hash algorithm, such as SHA256, SHA224, SHA238, SHA512, etc.

[0097] It should be understood that the data fingerprinting algorithm used by the electronic device to calculate the first data fingerprint and the second data fingerprint should be the same, so as to ensure the correctness when querying the corresponding physical address based on the data fingerprint.

[0098] Unlike the first data fingerprint, since the physical address of the memory data block needs to be updated later, the electronic device can insert the correspondence between the second data fingerprint and the second physical address into the FP-PA mapping table. This way, when performing data deduplication next time, the physical address corresponding to the data fingerprint can be queried from the new FP-PA mapping table.

[0099] 309. Insert the second data fingerprint and the corresponding second physical address into the FP-PA mapping table.

[0100] The electronic device inserts the second data fingerprint and the corresponding second physical address into the FP-PA mapping table to update the FP-PA mapping table, thereby ensuring the accuracy of file deduplication.

[0101] 310. Map the FP-PA mapping table to the user space of the first application.

[0102] In some embodiments, an electronic device may map an FP-PA mapping table to the user space of a first application via direct mapping.

[0103] For example, the memory mapping function mmap can be used to map the FP-PA mapping table to user space. In this case, the FP-PA mapping table is shared between user space and kernel space, which can increase the speed of searching for the physical address corresponding to a data block in user space.

[0104] In this way, the FP-PA mapping table can be shared in user space and kernel space, allowing applications to identify duplicate data in user space without having to pass the duplicate data itself to kernel space, thus avoiding additional copy overhead and improving the efficiency of file deduplication.

[0105] It should be understood that, provided that self-consistency is satisfied, some steps in steps 301 to 310 may be deleted, not performed, replaced, or interchanged, etc., and the embodiments of this application do not limit this.

[0106] File systems typically improve performance through defragmentation or garbage collection (GC). These operations involve moving large amounts of physical data blocks, essentially moving data from physical address A to physical address B. After deduplication of write I / O from applications, multiple logical addresses of a data block may correspond to a single physical address. Therefore, after defragmentation or garbage collection of that physical address, users accessing other logical addresses of that physical address may encounter data read errors. The following section will discuss this further. Figure 3 This paper introduces a technical solution to this technical problem.

[0107] Figure 3 This is a schematic diagram of the mapping relationship provided in the embodiments of this application. For example... Figure 3 As shown, the electronic device stores an L2P mapping table and a physical address to logical address (P2L) mapping table.

[0108] See Figure 3 In (a) of the diagram, before file deduplication, logical addresses and physical addresses can have a one-to-one correspondence. For example, in the L2P mapping table, the logical address of data block 1 is L1, and its corresponding physical address is P1; the logical address of data block 2 is L2, and its corresponding physical address is L2. Similarly, in the P2L mapping table, the physical address of data block 1 is P1, and its corresponding logical address is L1; the physical address of data block 2 is P2, and its corresponding logical address is L2. The physical address of data block 3 is P3, and its logical address is free, meaning that data block 3 has not been used and no data is stored at that physical address P3.

[0109] See Figure 3In (b) of the example, after file deduplication, the L2P mapping table may show multiple logical addresses corresponding to a single physical address. For example, in the L2P mapping table, the logical address of data block 1 is L1, and its corresponding physical address is P1. After data deduplication, the logical address of data block 2, L2, also corresponds to the physical address P1. However, in the P2L mapping table, P1 corresponds to L1, and P2 also corresponds to L2. Therefore, when garbage collecting P1, the system moves the data from P1 to P3. According to the P2L mapping table, the system can only be notified to change the physical address corresponding to L1 in the L2P mapping table to P3, not to change the physical address corresponding to L2 in the L2P mapping table to P3, because P1 does not correspond to L2 in the P2L mapping table. At this point, when a user accesses L2, since L2 corresponds to P1, but P1 has already been garbage collected, a data read error will occur.

[0110] It should be understood that the L2P mapping table can also be represented by logical block number - physical block number, and the P2L mapping table can also be represented by physical block number - logical block number. This application embodiment does not limit the representation.

[0111] To address the issue of data read errors that occur after file deduplication and defragmentation or garbage collection, this application divides the file system's disk space into multiple regions. For example, N 4KB data blocks can be considered as one region. Each region may contain metadata, which can include a target inode number. This target inode can be read from the disk based on its number. The target inode stores a target P2L mapping table, which records the correspondence between all physical addresses and logical addresses within that region. Therefore, when physical addresses within a region are reclaimed, the target P2L mapping table can be accessed. Based on the correspondence in the target P2L mapping table, multiple logical addresses corresponding to a physical address can be determined, allowing for the updating of the physical addresses corresponding to these multiple logical addresses and preventing data read errors.

[0112] In some embodiments, the value of N can be 256, 512, etc., and this application does not limit the specific value of N.

[0113] For example, after deduplication, L1 corresponds to P1, and L2 corresponds to P1. Then, when garbage collecting is performed on P1, for instance, moving data from P1 to P3, the target P2L mapping table is obtained by accessing the inode in the metadata. Based on this target P2L mapping table, it is known that P1 corresponds to L1, and P1 also corresponds to L2. Therefore, the system can be notified to modify the L2P mapping table. See also... Figure 3In (c), after the L2P mapping table is modified, L1 corresponds to P3, and L2 also corresponds to P3. After garbage collection, in the P2L mapping table, P1 is reclaimed and becomes free, P3 corresponds to L1, and P3 also corresponds to L2. In this way, after garbage collection, no data reading errors will occur whether the user accesses L1 or L2.

[0114] In this technical solution, the disk is divided into multiple regions, each with its own metadata section. This metadata section stores a P2L mapping table containing all the correspondences between physical addresses and logical addresses within the region. Based on this, when a physical address in a region is reclaimed, the P2L mapping table can be accessed. According to the correspondence in the P2L mapping table, multiple logical addresses corresponding to a physical address can be determined, allowing for the updating of the physical addresses corresponding to these multiple logical addresses and preventing data read errors.

[0115] Figure 4 This is a schematic flowchart illustrating a data deduplication method provided in an embodiment of this application. Figure 4 As shown, the method 400 may include steps 410 to 450. The method 400 can be applied to electronic devices.

[0116] 410, The electronic device obtains first information from the application; the first information includes the write I / O size and the page offset to be written.

[0117] In some embodiments, the electronic device can obtain initial information about the application through a system interface. For example, the electronic device intercepts POSIX functions of the first application using the LD_PRELOAD flag to obtain the write I / O size and page offset generated by the first application.

[0118] It should be understood that the write I / O size refers to the size of the data written by the application, and the page offset can be understood as the position corresponding to the application performing the write I / O. For example, if an application needs to write 4096 bytes of data from the 4097th byte of a file, then the write I / O size is 4096 bytes, and the page offset is the 4097th byte.

[0119] 420. The electronic device obtains the first memory data block based on the first information.

[0120] For example, when the write I / O size is large, the electronic device can divide or trim the write I / O size according to the page size and the page offset to obtain a first memory data block.

[0121] It should be understood that splitting or trimming the write I / O size can be interpreted as dividing the write I / O size into multiple memory data blocks.

[0122] It should be understood that the page size and the page offset to be written can be found in the description above, and will not be repeated here for the sake of brevity.

[0123] In some embodiments, when the write I / O size is less than or equal to a page size (e.g., 4KB), it is not necessary to split or trim the write I / O size; the write I / O size can be directly treated as the first memory data block.

[0124] 430, The electronic device calculates the first data fingerprint of the first memory data block.

[0125] It should be understood that the method by which the electronic device calculates the first data fingerprint of the first memory data block can be found in the relevant description of step 303.

[0126] 440. The electronic device determines whether a first physical address corresponding to the first data fingerprint exists.

[0127] In some embodiments, an electronic device may determine whether a first physical address corresponding to a first data fingerprint exists by querying a mapping table of data fingerprints and physical addresses.

[0128] It should be understood that the mapping table between the data fingerprint and the physical address can also be in other forms, such as an array, a skip list, a B-tree, etc.

[0129] 450. When it is determined that there is a first physical address corresponding to the first data fingerprint, the electronic device updates the mapping relationship between the first memory data block and the first physical address.

[0130] For example, when the electronic device determines that the first physical address corresponding to the first data fingerprint exists, it means that the data in the first memory data block has been stored and is duplicated. Then the electronic device can update the mapping relationship between the first memory data block and the first physical address, so that it does not need to store it repeatedly.

[0131] In one embodiment of this application, the electronic device can obtain first information of the application and obtain a first memory data block based on the first information. The first memory data block can be multiple memory data blocks. The device can also calculate the data fingerprint of the first memory data block. When it is determined that a physical address corresponding to the data fingerprint exists, it can be determined that the data in the memory data block already exists. Then, the mapping relationship between the first memory data block and the first physical address can be updated directly without passing the data of the first memory data block itself. This allows the technical solution to achieve zero or near-zero copy overhead for duplicate data when performing data deduplication, thereby improving the speed of the application's write process.

[0132] It should be understood that this application does not limit the execution order of steps 410 to 450. In some embodiments, some steps of steps 410 to 450 may not be executed, or may be replaced by other steps, etc.

[0133] In some embodiments, when the first memory data block comprises multiple page-aligned memory data blocks, the electronic device calculates a first data fingerprint of the first memory data block, including: the electronic device performs a first calculation on the first memory data block in the user space of the application to obtain the first data fingerprint.

[0134] For example, when an electronic device needs to divide a write I / O size into a first memory data block, this first memory data block may include multiple page-aligned memory data blocks. For instance, if the write I / O size is 12KB, it can be divided into three page-aligned memory data blocks. In this case, the electronic device can perform a first calculation on each memory data block within the application's user space to obtain a data fingerprint for each memory data block.

[0135] This first calculation can be performed using a data fingerprinting algorithm, as described above.

[0136] In one embodiment of this application, when the first memory data block includes a page-aligned memory data block, the electronic device can directly calculate the corresponding first data fingerprint of the first memory data block in the user space of the application. This facilitates the electronic device to query the corresponding physical address based on the first data fingerprint, without having to transfer the actual data to the kernel space, thus achieving zero data copy.

[0137] In some embodiments, when the first memory data block comprises a plurality of page-aligned memory data blocks and a non-page-aligned memory data block, the method 400 may further include the following before the electronic device calculates the first data fingerprint of the first memory data block:

[0138] The electronic device transfers data from the user space of the application to the kernel space of the electronic device in a non-page-aligned memory block; the electronic device then converts the non-page-aligned memory block into a page-aligned memory block in the kernel space.

[0139] For example, an electronic device transfers a non-page-aligned memory data block from the user space of an application to the kernel space, and the page cache layer in the kernel space pads the non-page-aligned memory data block with zeros to convert it into a page-aligned memory data block, and then calculates the data fingerprint of the first memory data block.

[0140] It should be understood that other layers or modules in the kernel space can also perform page cache layer operations, and this application embodiment does not limit this.

[0141] In other embodiments, when an electronic device transmits non-page-aligned memory data blocks, it may also simultaneously calculate the data fingerprint of page-aligned memory data blocks, which is not limited in the embodiments of this application.

[0142] In one embodiment of this application, when the first memory data block includes a page-aligned memory data block and a non-page-aligned memory data block, the electronic device can convert the non-page-aligned memory data block into a page-aligned memory data block in the kernel space. Thus, the electronic device only needs to transfer the data in a non-page-aligned memory data block from user space to kernel space, which greatly reduces the amount of data transferred and basically achieves zero data copying.

[0143] In some embodiments, the first information further includes a file descriptor; before the electronic device updates the mapping relationship between the first memory data block and the first physical address, the method 400 may further include:

[0144] The electronic device transfers the file descriptor, the page offset to be written, and the first physical address from the application's user space to the electronic device's kernel space.

[0145] In this embodiment, the electronic device transfers the file descriptor, the written page offset, and the first physical address from user space to kernel space, thereby eliminating the need to copy the data of the first memory data block itself to kernel space and reducing data copying.

[0146] In some embodiments, updating the mapping relationship between the first memory data block and the first physical address by the electronic device includes: updating the mapping relationship between the first memory data block and the first physical address in the kernel space according to the file descriptor, the page offset written, and the first physical address.

[0147] It should be understood that the electronic device updates the mapping relationship between the first memory data block and the first physical address in the kernel space based on the file descriptor, the page offset written, and the first physical address. This can be referred to in the relevant description above, and will not be repeated here for the sake of brevity.

[0148] In this embodiment, the electronic device can update the mapping relationship between the first memory data block and the first physical address based on the file descriptor, the page offset written, and the first physical address. This eliminates the need to copy the data of the first memory data block itself to the kernel space, reducing data copying and improving the speed of the write process.

[0149] In some embodiments, the electronic device stores a first mapping table, which is a mapping table of data fingerprints and physical addresses; determining that there exists a first physical address corresponding to the first data fingerprint includes: querying the first physical address from the first mapping table based on the first data fingerprint.

[0150] For example, the first mapping table is the FP-PA mapping table mentioned above. When the electronic device can query the corresponding first physical address from the FP-PA mapping table based on the first data fingerprint, it can determine the first physical address stored in the first data fingerprint, thereby determining that the data in the data block corresponding to the first data fingerprint is duplicated.

[0151] In some embodiments, the method 400 may further include:

[0152] When it is determined that there is no first physical address corresponding to the first data fingerprint, the electronic device allocates a first physical data block for the first memory data block; the electronic device writes the data in the first memory data block into the first physical data block.

[0153] For example, see Figure 1 When it is determined that there is no first physical address corresponding to the first data fingerprint, it can be determined that the data in the first memory data block is not duplicate data. Then, the allocation module 135 in the electronic device can allocate a first physical data block for storing the data in the first memory data block. The write module 134 in the electronic device can write the data in the first memory data block into the first physical data block.

[0154] In this embodiment of the application, when it is determined that there is no first physical address corresponding to the first data fingerprint, it can be determined that the data in the first memory data block is not duplicated. At this time, the electronic device can allocate a new physical data block for storing its data.

[0155] In some embodiments, the physical address corresponding to the first physical data block is the second physical address, and the method 400 may further include:

[0156] Electronic devices store the mapping relationship between a first memory data block and a second physical address.

[0157] Based on the embodiments of this application, an electronic device can save the mapping relationship between a first memory data block and a second physical address where the data is actually stored, thereby determining the actual storage location of the first memory data block.

[0158] In some embodiments, the method 400 may further include:

[0159] The electronic device calculates the second data fingerprint of the first physical data block; the electronic device saves the second data fingerprint and the second physical address to the first mapping table.

[0160] For example, see Figure 1 The computing module 136 in the electronic device can calculate the second data fingerprint of the first physical data block and save the second data fingerprint and the second physical address to the FP-PA mapping table 140a to update the first mapping table, thereby ensuring the accuracy of data deduplication in the next operation.

[0161] In some embodiments, the method 400 may further include:

[0162] The electronic device maps the first mapping table from the kernel space of the electronic device to the user space of the application.

[0163] For example, see Figure 1 The electronic device maps the FP-PA mapping table 140a to the user space of the application through direct mapping, allowing the user-space deduplication module 120 to read the FP-PA mapping table in read-only mode. This ensures that the first mapping table is shared between user space and kernel space, thereby increasing the speed of searching for the physical address corresponding to a data block in user space. Furthermore, the first mapping table mapped to user space is read-only, ensuring that the first mapping table in kernel space and the first mapping table in user space are identical, preventing one application from modifying the first mapping table in user space and affecting other applications.

[0164] This application also provides an electronic device, including one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by one or more processors, cause a data deduplication method as described in any of the possible implementations above to be executed.

[0165] This application also provides a chip, which includes a processor and a communication interface. The communication interface is used to receive signals and transmit the signals to the processor. The processor processes the signals so that the data deduplication method described in any of the possible implementations above is executed.

[0166] This embodiment also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the data deduplication method described in the above embodiment to be performed.

[0167] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the data deduplication method described in the above embodiment.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0173] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for data deduplication, characterized in that, The method is applied to an electronic device, and the method includes: The electronic device acquires first information about the application; the first information includes write I / O size and write page offset; The electronic device obtains a first memory data block based on the first information; When the first memory data block includes multiple page-aligned memory data blocks and one non-page-aligned memory data block, the electronic device transfers the non-page-aligned memory data block from the user space of the application to the kernel space of the electronic device. The electronic device converts the non-page-aligned memory data block into a page-aligned second memory data block in the kernel space; The electronic device calculates the first data fingerprint of the first memory data block, including: The electronic device calculates the data fingerprint of the second memory data block in the kernel space, and calculates the data fingerprint of the third memory data block that is page-aligned in the first memory data block in the user space of the application. The electronic device determines whether a first physical address corresponding to the first data fingerprint exists, including: Determine in kernel space whether a physical address corresponding to the data fingerprint of the second memory data block exists, and determine in user space whether a physical address corresponding to the data fingerprint of the third memory data block exists. When it is determined that a first physical address corresponding to the target data fingerprint exists, the electronic device updates the mapping relationship between the memory data block corresponding to the target data fingerprint and the first physical address, wherein the target data fingerprint is at least one of the data fingerprint of the second memory data block or the data fingerprint of the third memory data block.

2. The method according to claim 1, characterized in that, The electronic device obtains a first memory data block based on the first information, including: The electronic device divides the write I / O size according to the page size and the page offset to obtain the first memory data block.

3. The method according to claim 1 or 2, characterized in that, The first information also includes a file descriptor; before the electronic device updates the mapping relationship between the memory data block corresponding to the target data fingerprint and the first physical address, the method further includes: The electronic device transfers the file descriptor, the written page offset, and the first physical address from the application's user space to the electronic device's kernel space.

4. The method according to claim 3, characterized in that, The electronic device updates the mapping relationship between the memory data block corresponding to the target data fingerprint and the first physical address, including: Based on the file descriptor, the written page offset, and the first physical address, the electronic device updates the mapping relationship between the memory data block corresponding to the target data fingerprint and the first physical address in the kernel space.

5. The method according to claim 1, 2, or 4, characterized in that, The electronic device stores a first mapping table, which is a mapping table between data fingerprints and physical addresses; determining that a first physical address corresponding to the first data fingerprint exists includes: The first physical address is retrieved from the first mapping table based on the first data fingerprint.

6. The method according to claim 1, 2, or 4, characterized in that, The method further includes: When it is determined that there is no first physical address corresponding to the target data fingerprint, the electronic device allocates a first physical data block for the memory data block corresponding to the target data fingerprint; The electronic device writes the data in the memory data block corresponding to the target data fingerprint into the first physical data block.

7. The method according to claim 6, characterized in that, The physical address corresponding to the first physical data block is the second physical address, and the method further includes: The electronic device stores the mapping relationship between the memory data block corresponding to the target data fingerprint and the second physical address.

8. The method according to claim 7, characterized in that, The method further includes: The electronic device calculates a second data fingerprint of the first physical data block; The electronic device saves the second data fingerprint and the second physical address to the first mapping table.

9. The method according to claim 8, characterized in that, The method further includes: The electronic device maps the first mapping table from the kernel space of the electronic device to the user space of the application.

10. An electronic device, characterized in that, include: Modules for implementing the data deduplication method as described in any one of claims 1-9.

11. An electronic device, characterized in that, include: One or more processors; One or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by one or more processors, cause the data deduplication method as described in any one of claims 1-9 to be performed.

12. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being used to receive signals and transmit the signals to the processor, the processor processing the signals such that the data deduplication method as described in any one of claims 1-9 is executed.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the data deduplication method as described in any one of claims 1-9 to be performed.

Images