SSD Limited Window Data Deduplication Identification Method, Apparatus, and Computer Equipment

Abstract

This application relates to a method, apparatus, computer device, and storage medium for SSD limited-window data deduplication identification. The method includes: if the command is a write command, allocating a write buffer and receiving data from the host; dividing the write command into data segments; calculating a data digest for each data segment; determining whether the data digests of each data segment of the write command are consistent with the data digests in the read data deduplication window; if the data digests are consistent, further determining whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window; if the data content is also consistent, sending the corresponding read data segment logical address and write data segment logical address to a write deduplication application module, which writes the data to the NAND flash memory using a mapping table copy method instead of writing the data directly. This invention can reduce data writes, improve SSD reliability, and significantly improve the performance of intra-disk data copying.

Description

Technical Field

[0001] This invention relates to the field of solid-state drive technology, and in particular to a method, apparatus, computer device, and storage medium for deduplication and identification of SSD limited-window data. Background Technology

[0002] With the development of solid-state drive (SSD) technology, SSDs have been widely used in various applications. In the PC market, they have gradually replaced traditional HDDs, providing users with a better experience in terms of reliability and performance. Internally, SSDs use NAND flash memory as the data storage medium. Host access to the SSD is based on logical addresses (LBAs), and the SSD maintains a logical-to-physical address mapping table (L2P table) to indicate the physical address where the corresponding logical address data is stored. When writing data, a physical storage address is allocated for the logical address data written by the host, and the corresponding L2P table is updated. When reading data, the physical address corresponding to the logical address is looked up in the L2P table, and the data is then retrieved and returned to the host.

[0003] Currently, existing SSDs contain a significant amount of duplicate data. These duplicates have different logical and physical addresses but identical data content, consuming additional NAND flash memory. During a typical host file copy operation, data needs to be read from the source logical address to the host and then written to a new logical address (destination address). Within a limited time window, a large amount of duplicate data exists. This process requires multiple read / write operations from the NAND flash, and the data needs to pass through the bus multiple times, severely limiting performance. Furthermore, writing this duplicate data consumes additional NAND flash memory, shortening the SSD's lifespan. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, and storage medium for deduplication and identification of SSD limited window data in response to the above-mentioned technical problems.

[0005] A method for deduplication and identification of SSD limited window data, the method comprising:

[0006] Obtain the command issued by the host and determine whether the command is a read command;

[0007] If the command is a read command, the data is read through the normal path. The read command is divided into data segments, and a data digest is calculated for each data segment.

[0008] Add the logical address of the read data segment, the data digest, and the data content to the read deduplication queue, move the read data deduplication window, and keep the number of records in the window constant.

[0009] If the command is not a read command, continue to determine whether the command is a write command;

[0010] If the command is a write command, a write buffer is allocated and data is received from the host. The write command is divided into data segments, and a data digest is calculated for each data segment.

[0011] Determine whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window;

[0012] If the data digests are consistent, it is further determined whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window;

[0013] If the data content is also consistent, the corresponding read data segment logical address and write data segment logical address are sent to the write deduplication application module, which replaces the write data with a mapping table copy method to write to NAND.

[0014] In one embodiment, the step of dividing the read command into data segments and calculating a data digest for each data segment further includes:

[0015] The read command is divided into multiple data segments of a certain size. Each data segment contains the logical address of the read data segment, a data digest, and the data content.

[0016] Maintain the read data segments accessed by the most recent host in order of time, forming a read deduplication queue.

[0017] In one embodiment, after the step of maintaining the most recently accessed read data segments in chronological order to form a read deduplication queue, the method further includes:

[0018] In the read data deduplication window, only a certain number of recently accessed read data segments are kept, while older records are discarded to reduce resource overhead and the cost of hit checks.

[0019] In one embodiment, the step of further determining whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window if the data digests are consistent further includes:

[0020] If the data digests match, the data content of the write data segment is XORed with the data content of the matched read data segment, and the result is checked to see if it is 0.

[0021] If the result is 0, the data is considered duplicated. The logical address of the write data end and the logical address of the matched read data segment are sent to the write deduplication application module. The write deduplication application module copies the mapping table of the read data segment logical address to the mapping table of the write data segment logical address to complete the data copy.

[0022] A device for deduplicating SSD data within a limited window, the device comprising:

[0023] The first judgment module is used to obtain the command issued by the host and determine whether the command is a read command;

[0024] The read deduplication window module is used to complete data reading according to the normal path if the command is a read command, divide the read command into data segments, calculate the data digest for each data segment, add the logical address of the read data segment, the data digest and the data content to the read deduplication queue, move the read data deduplication window and keep the number of records in the window constant.

[0025] The second judgment module is used to further determine whether the command is a write command if the command is not a read command.

[0026] The write deduplication identification module is used to allocate a write buffer and receive data from the host if the command is a write command, divide the write command into data segments, and calculate the data digest for each data segment.

[0027] The third judgment module is used to determine whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window;

[0028] The fourth judgment module is used to further determine whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window if the data digest is consistent; if the data content is also consistent, the corresponding read data segment logical address and write data segment logical address are sent to the write deduplication application module.

[0029] The write deduplication application module is used to replace writing data to NAND by copying the mapping table.

[0030] In one embodiment, the read deduplication window module is further configured to:

[0031] The read command is divided into multiple data segments of a certain size. Each data segment contains the logical address of the read data segment, a data digest, and the data content.

[0032] Maintain the read data segments accessed by the most recent host in order of time, forming a read deduplication queue.

[0033] In one embodiment, the read deduplication window module is further configured to:

[0034] In the read data deduplication window, only a certain number of recently accessed read data segments are kept, while older records are discarded to reduce resource overhead and the cost of hit checks.

[0035] In one embodiment, the fourth determination module is further configured to:

[0036] If the data digests match, the data content of the write data segment is XORed with the data content of the matched read data segment, and the result is checked to see if it is 0.

[0037] If the result is 0, the data is considered duplicated. The logical address of the write data end and the logical address of the matched read data segment are sent to the write deduplication application module. The write deduplication application module copies the mapping table of the read data segment logical address to the mapping table of the write data segment logical address to complete the data copy.

[0038] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0040] The aforementioned SSD limited-window data deduplication identification method, device, computer equipment, and storage medium maintain a certain depth of read data deduplication window internally, containing the logical address, data digest, and data content of the most recently read data segment. When data is written to the host, the SSD internally generates a data digest in real time for the written data segment and compares it with the data digest in the current read data deduplication window. For data digest matching scenarios, the corresponding data content is further compared. This allows the SSD to autonomously identify duplicate data within the limited window, thereby reducing data writes and improving SSD reliability through methods such as mapping table copying, while also significantly improving the performance of intra-disk data copying. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a typical SSD read / write process in traditional technology;

[0042] Figure 2 This is a schematic diagram of the host file data copying process in traditional technology;

[0043] Figure 3 This is a flowchart illustrating a method for deduplication and identification of SSD limited window data in one embodiment;

[0044] Figure 4This is a flowchart illustrating the SSD limited window data deduplication identification method in another embodiment;

[0045] Figure 5 This is a schematic diagram of a finite window deduplication module introduced in one embodiment;

[0046] Figure 6 This is a schematic diagram illustrating the maintenance of the deduplication window in one embodiment;

[0047] Figure 7 This is a schematic diagram illustrating the specific implementation of the SSD limited window data deduplication identification method in one embodiment;

[0048] Figure 8 This is a structural block diagram of an SSD limited window data deduplication and identification device in one embodiment;

[0049] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] refer to Figure 1 The typical SSD read / write process shown includes the following steps: The host sends a command to the SSD hardware module; the SSD hardware module (PCIe / NVMe) receives the command and forwards it to the firmware module for processing; the SSD firmware front-end module divides the command into mapping units (typically 4KB); it submits the operation request to the buffer management module, allocating read / write buffers; if it is a write command, it establishes data transmission with the host based on the allocated buffer and notifies the host of command completion after data transmission is complete; if it is a read command, it submits the operation request to the mapping table management module; the mapping table management module is responsible for allocating the corresponding physical address based on the logical address (write command) or converting the logical address into a NAND physical address (read command); it submits the operation request to the back-end module, which initiates a NAND read / write request based on the physical address; it waits for the NAND read / write operation request to complete; if it is a read command, after the data is Ready, it starts the data transmission from the NAND Cache Register to the host.

[0052] refer to Figure 2 The process of copying host file data shown includes the following implementation steps:

[0053] S0: Obtain the logical addresses of the source and destination data to be copied.

[0054] S1: The host reads source data fragment 1.

[0055] S2: The SSD converts LBA to LPA; queries the physical address of the data to be read; and initiates a read operation at the corresponding physical address.

[0056] S3: The SSD returns data to the host.

[0057] S4: The host writes data fragment 1 to the destination address.

[0058] S5: Convert LBA to LPA; allocate the physical address of the data to be written and update the mapping table; initiate the write to the corresponding physical address.

[0059] Repeat steps S1-S5 until all data fragments have been copied.

[0060] In the above process, the source data needs to be read from the SSD / NAND sequentially, transferred to the host memory via the bus (PCIe), then transferred to the SSD via the bus, and then written to the NAND. This involves many operations, which greatly restricts the performance of data copying. At the same time, the repeated writing of data to the NAND consumes the lifespan of the SSD.

[0061] Based on this, the present invention proposes a method for identifying duplicate data in SSD limited window, which aims to identify such duplicate data characteristics in order to optimize SSD performance and reduce SSD write volume.

[0062] In one embodiment, such as Figure 3 As shown, a method for deduplication and identification of SSD limited window data is provided, the method including:

[0063] Step 302: Obtain the command issued by the host and determine whether the command is a read command;

[0064] Step 304: If the command is a read command, the data is read according to the normal path. The read command is divided into data segments, and a data digest is calculated for each data segment.

[0065] Step 306: Add the logical address of the read data segment, the data digest, and the data content to the read deduplication queue, move the read data deduplication window, and keep the number of records in the window constant.

[0066] Step 308: If the command is not a read command, continue to determine whether the command is a write command;

[0067] Step 310: If the command is a write command, allocate a write buffer and receive data from the host, divide the write command into data segments, and calculate the data digest for each data segment.

[0068] Step 312: Determine whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window;

[0069] Step 314: If the data digest is consistent, further determine whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window;

[0070] Step 316: If the data content is also consistent, the corresponding read data segment logical address and write data segment logical address are sent to the write deduplication application module. The write deduplication application module replaces the write data with the write data in the NAND by copying the mapping table.

[0071] This embodiment provides a method for identifying duplicate data in SSDs with limited command windows. This method can effectively identify data redundancy in scenarios with limited command windows, providing feasibility for further performance optimization and lifespan improvement. Specifically, it includes the following steps:

[0072] First, obtain the command issued by the host and determine whether the command is a read command; if the command is a read command, complete the data reading according to the normal path, divide the read command into data segments, and calculate the data digest for each data segment.

[0073] refer to Figure 5 The diagram shows a finite window deduplication module. A deduplication module has been added to the command processing path, which is further divided into three sub-modules: reading the deduplication window, writing deduplication identification, and writing deduplication application.

[0074] The deduplication window is used to maintain information about the most recently read data segment inside the SSD: logical address, data digest, and data content, which is used to match it with newly written data segments.

[0075] Write deduplication identification is used to generate a data digest for newly written data segments on the host and compare it with the records stored in the read deduplication window queue to determine whether the digests are the same. If the digests are the same, the data is further compared to confirm whether the data is completely consistent. For write commands where both the digest and the data content are consistent, the logical address and the logical address information of the hit read data segment are sent to the write deduplication application module.

[0076] The write deduplication module is used to copy the mapping table of the read logical address to the mapping table of the write logical address based on the received logical addresses of the write data segment and the read data segment, thereby completing the data copy.

[0077] In this embodiment, the SSD internally maintains a read data deduplication window with a certain depth, containing the logical address, data digest, and data content of the most recently read data segment. When the host writes data, the SSD internally generates a data digest for the written data segment in real time and compares it with the data digest in the current read data deduplication window. For data digest matching scenarios, the corresponding data content is further compared. This allows the SSD to autonomously identify duplicate data within a limited window, thereby reducing data writes and improving SSD reliability through methods such as mapping table copying, while also significantly improving the performance of intra-disk data copying.

[0078] In one embodiment, such as Figure 4 As shown, a method for deduplication and identification of SSD limited window data is provided. This method further includes dividing read commands into data segments and calculating a data digest for each segment.

[0079] Step 402: Divide the read command into multiple data segments of a certain size. Each data segment contains the logical address of the read data segment, the data digest, and the data content.

[0080] Step 404: Maintain the read data segments accessed by the most recent host in order of time from oldest to newest, and form a read deduplication queue;

[0081] Step 406: In the read data deduplication window, only a certain number of recently accessed read data segments are kept, while older records are discarded to reduce resource overhead and the cost of hit checks.

[0082] refer to Figure 6 The diagram shown illustrates the read deduplication window maintenance process. Specifically, the read command is first divided into data segments of a certain size (customizable, such as 4KB / 8KB / ...). Each data segment contains the following information:

[0083] Read the logical address of the data segment, which corresponds to the starting logical address (LBA) of the data segment;

[0084] Data digest, the data digest of the corresponding data segment, can be roughly compared using common algorithms such as SHA / MD5... (fast, but with a certain probability of false positives);

[0085] Data content, the original content of this data segment, used for precise comparison (slower speed).

[0086] Then, based on the time sequence from oldest to newest, maintain the read data segments accessed by the most recently accessed host to form a read deduplication queue.

[0087] Finally, in the read data deduplication window, only a certain number of recently accessed read data segments are kept (the number can be customized, such as 4 / 8 / 16...), while older records are discarded to reduce resource overhead and the cost of hit checks.

[0088] In this embodiment, in a typical data copying scenario, N data segments are typically read and N data segments are written. Therefore, write commands will generally hit the N most recent read command data segments. Based on this N, the size of the read deduplication window (the number of data segments) can be adjusted to improve efficiency.

[0089] In one embodiment, a method for identifying duplicate data within a limited window in an SSD is provided. This method further includes: if the data digests match, performing an XOR operation between the data content of the write data segment and the data content of the matched read data segment, and determining if the result is 0; if the result is 0, determining that the data is duplicated, and sending the logical address of the write data segment and the logical address of the matched read data segment to a write deduplication application module; the write deduplication application module then copies the mapping table of the read data segment logical addresses to the mapping table of the write data segment logical addresses to complete the data copy.

[0090] In this embodiment, firstly, the digest of the written data segment is compared with the data digest in the current read data deduplication window. If they match, further data matching is performed. For scenarios where the data digest matches, the corresponding read and write data are further XORed. If the result is 0, data duplication is confirmed.

[0091] The following is a specific example for illustration. Figure 7 The diagram shown illustrates the specific execution flow. The method includes:

[0092] Step 7.1: Obtain new commands from the host.

[0093] Step 7.2: Determine if it is a read command; if yes, continue to step 7.3; if no, jump to step 7.7.

[0094] Step 7.3: Complete the data reading according to the normal path.

[0095] Step 7.4: Divide the read command into data segments and calculate the data digest for each data segment.

[0096] Step 7.5: Add (data segment logical address, data digest, data content) to the read deduplication queue and move the read data deduplication window to keep the number of records in the window constant.

[0097] Step 7.6: Command completed.

[0098] Step 7.7: Determine if it is a write command; if yes, continue to step 7.8; if no, process according to the normal path.

[0099] Step 7.8: Allocate a write buffer and receive data from the host.

[0100] Step 7.9: Divide the write command into data segments and calculate the data digest for each data segment.

[0101] Step 7.10: Compare the data digests of each data segment of the write command with the data digests in the read data deduplication window.

[0102] Step 7.11: Determine if they are equal; if yes, continue to step 7.12; if no, write the data to NAND according to the normal path.

[0103] Step 7.12: XOR the content of the write data segment with the content of the hit read data segment.

[0104] Step 7.13: Determine if the result is 0; if yes, continue to step 7.14; if no, write the data to NAND according to the normal path.

[0105] Step 7.14: Data duplication. Send the corresponding read data segment logical address and write data segment logical address to the write deduplication application module.

[0106] Step 7.15: The write deduplication application module replaces the write data with NAND by copying the mapping table according to the logical address of the read and write data segments.

[0107] Step 7.16: Command completed.

[0108] It should be understood that, although Figure 1-7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-7 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0109] In one embodiment, such as Figure 8 As shown, an SSD limited window data deduplication identification device 800 is provided, the device comprising:

[0110] The first judgment module 801 is used to obtain the command issued by the host and determine whether the command is a read command;

[0111] The read deduplication window module 802 is used to complete data reading according to the normal path if the command is a read command, divide the read command into data segments, calculate the data digest for each data segment, add the logical address of the read data segment, the data digest and the data content to the read deduplication queue, move the read data deduplication window and keep the number of records in the window constant.

[0112] The second judgment module 803 is used to further determine whether the command is a write command if the command is not a read command.

[0113] The write deduplication identification module 804 is used to allocate a write buffer and receive data from the host if the command is a write command, divide the write command into data segments, and calculate the data digest for each data segment.

[0114] The third judgment module 805 is used to determine whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window;

[0115] The fourth judgment module 806 is used to further determine whether the data content of each data segment of the write command is consistent with the data content in the read data deduplication window if the data digest is consistent; if the data content is also consistent, the corresponding read data segment logical address and write data segment logical address are sent to the write deduplication application module.

[0116] Write deduplication application module 807, which is used to replace writing data to NAND by copying the mapping table.

[0117] In one embodiment, the deduplication window reading module 802 is further configured to:

[0118] The read command is divided into multiple data segments of a certain size. Each data segment contains the logical address of the read data segment, a data digest, and the data content.

[0119] Maintain the read data segments accessed by the most recent host in order of time, forming a read deduplication queue.

[0120] In one embodiment, the deduplication window reading module 802 is also used for:

[0121] In the read data deduplication window, only a certain number of recently accessed read data segments are kept, while older records are discarded to reduce resource overhead and the cost of hit checks.

[0122] In one embodiment, the fourth determination module 806 is further configured to:

[0123] If the data digests match, the data content of the write data segment is XORed with the data content of the matched read data segment, and the result is checked to see if it is 0.

[0124] If the result is 0, the data is considered duplicated. The logical address of the write data end and the logical address of the matched read data segment are sent to the write deduplication application module. The write deduplication application module copies the mapping table of the read data segment logical address to the mapping table of the write data segment logical address to complete the data copy.

[0125] For specific limitations on SSD limited-window data deduplication identification devices, please refer to the limitations on SSD limited-window data deduplication identification methods mentioned above, which will not be repeated here.

[0126] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a limited-window data deduplication identification method for SSDs.

[0127] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the various method embodiments described above.

[0129] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for SSD limited window data deduplication identification, the method comprising: obtaining a command issued by a host and determining whether the command is a read command; if the command is a read command, completing data reading in a normal path, dividing the read command into data segments, and calculating data digest for each data segment; adding read data segment logical address, data digest, and data content to a read deduplication queue, moving a read data deduplication window, and dynamically adjusting the size of the window based on the number N of read and write operations in a data copy scenario to match the number of data segments of the last N read operations; if the command is not a read command, continuing to determine whether the command is a write command; if the command is a write command, allocating a write buffer and receiving data from the host, dividing the write command into data segments, and calculating data digest for each data segment; determining whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window; if the data digests are consistent, performing XOR operation on the data content of the write data segment and the data content of the hit read data segment, and determining whether the result is 0; if the result is 0, determining that the data is duplicated, and sending the logical address of the corresponding read data segment and the logical address of the write data segment to a write deduplication application module, which directly copies the physical address of the read data segment logical address to the write data segment logical address through a mapping table to replace the write data to be written to NAND.

2. The method of claim 1, wherein, The step of dividing the read command into data segments and calculating data digest for each data segment further comprises: dividing the read command into a plurality of data segments according to a certain size, each data segment containing read data segment logical address, data digest, and data content; maintaining read data segments accessed by the host in a time order from far to near to form a read deduplication queue.

3. The method of claim 2, wherein, The step of maintaining read data segments accessed by the host in a time order from far to near to form a read deduplication queue further comprises: in the read data deduplication window, only maintaining a certain number of recently accessed read data segments, and discarding further records to reduce resource overhead and the cost of hit checking.

4. A device for deduplicating and identifying SSD limited-window data, characterized in that, The SSD limited window data deduplication identification device comprises: a first determination module configured to obtain a command issued by a host and determine whether the command is a read command; a read deduplication window module configured to, if the command is a read command, complete data reading in a normal path, divide the read command into data segments, calculate data digest for each data segment, add read data segment logical address, data digest, and data content to a read deduplication queue, move a read data deduplication window, and dynamically adjust the size of the window based on the number N of read and write operations in a data copy scenario to match the number of data segments of the last N read operations; a second determination module configured to, if the command is not a read command, continue to determine whether the command is a write command; a write deduplication identification module configured to, if the command is a write command, allocate a write buffer and receive data from the host, divide the write command into data segments, and calculate data digest for each data segment. A third judging module, configured to judge whether the data digest of each data segment of the write command is consistent with the data digest in the read data deduplication window; A fourth judging module, configured to, if the data digests are consistent, perform XOR operation on the data content of the write data segment and the data content of the hit read data segment, and judge whether the result is 0; if the result is 0, judge that the data is duplicated, and send the corresponding read data segment logical address and write data segment logical address to the write deduplication application module; The write deduplication application module is configured to copy the physical address of the read data segment logical address to the write data segment logical address through the mapping table, and replace the write data to be written into the NAND.

5. The SSD finite window data deduplication identification apparatus of claim 4, wherein, The read deduplication window module is further configured to: divide the read command into a plurality of data segments according to a certain size, and each data segment contains a read data segment logical address, a data digest and data content; maintain the read data segment accessed by the host recently according to time from far to near, and form a read deduplication queue.

6. The SSD finite window data deduplication identification apparatus of claim 5, wherein, The read deduplication window module is further configured to: In the read data deduplication window, only a certain number of read data segments accessed recently are maintained, and the records farther away are discarded to reduce resource overhead and the cost of hit check.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 3.

Images