Data processing method, device, computer equipment and storage medium

Abstract

The present application relates to a data processing method, device, computer equipment and storage medium. The method includes: determining whether there is a fingerprint value matching the target fingerprint value in a preset data table; if not, performing a disk write operation on the target data and the target fingerprint value based on a preset calculation rule; if so, obtaining the flash translation layer address corresponding to the target fingerprint value, obtaining the target disk address based on the flash translation layer address and then reading the data on the target disk, and determining whether the data on the target disk is consistent with the target data; if not, reallocating the target physical address for the target data and writing the reallocated target physical address into a preset data table; if consistent, updating the target logical address corresponding to the target data into a target logical address linked list in a preset data table, and when receiving a data deduplication instruction, deleting the logical address corresponding to the data to be deleted from the preset data table. The use of this method can improve system performance.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] Data deduplication is a data reduction technology and a mainstream storage technology. To achieve better deduplication results, the industry currently uses software-based deduplication. However, enabling deduplication significantly impacts the entire software stack's ability to process I / O, resulting in a significant performance degradation. Summary of the Invention

[0003] Based on this, it is necessary to provide a data processing method, device, computer equipment and storage medium that can improve system performance in response to the above technical problems.

[0004] In order to solve the above technical problems, a first aspect is to provide a data processing method, which includes:

[0005] In response to receiving the target data sent by the host, calculating the target fingerprint value of the target data;

[0006] Obtain a preset data table and determine whether there is a fingerprint value matching the target fingerprint value in the preset data table;

[0007] If it does not exist, the target data and target fingerprint value are written to disk based on the preset calculation rules;

[0008] If it exists, obtain the flash translation layer address corresponding to the target fingerprint value, and obtain the target disk address based on the flash translation layer address;

[0009] Read the data on the target disk based on the target disk address, and determine whether the data on the target disk is consistent with the target data;

[0010] If they are inconsistent, reallocate the target physical address for the target data and write the reallocated target physical address into the preset data table;

[0011] If they are consistent, the target logical address corresponding to the target data is updated to the target logical address linked list corresponding to the target physical address in the preset data table;

[0012] In response to receiving a data deduplication instruction carrying information about data to be deleted, a logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from a preset data table.

[0013] In one embodiment, the preset data table includes a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores physical addresses corresponding to the fingerprint data and a logical address linked list formed by logical addresses pointing to each physical address.

[0014] In one embodiment, performing a write operation on the target data and the target fingerprint value based on a preset calculation rule includes:

[0015] Obtain target data and target fingerprint value;

[0016] Using the target fingerprint value as a seed, calculate the ASCII value of the target data, and assign a random number to the seed based on the ASCII value of the target data;

[0017] Get the number of disks that make up the array in the system;

[0018] The remainder is obtained by taking the number of disks constituting the array as the denominator and the random number as the numerator and multiplying the random number by the number of disks constituting the array.

[0019] Find the target data and the target disk corresponding to the target fingerprint value according to the remainder;

[0020] Download the target data and target fingerprint value to the target disk.

[0021] In one embodiment, flashing the target data and the target fingerprint value to the target disk further includes:

[0022] Obtain the target logical address and target physical address corresponding to the target data;

[0023] Update the target logical address and the target physically into the flash translation layer table;

[0024] In response to the target data and the target fingerprint value being flushed to the target disk, a successful flush instruction is generated, the target fingerprint value and the flash translation layer address corresponding to the target data are obtained, and the target fingerprint value and the flash translation layer address corresponding to the target data are inserted into a preset data table.

[0025] In one embodiment, obtaining a flash translation layer address corresponding to a target fingerprint value, and obtaining a target disk address based on the flash translation layer address includes:

[0026] Get the flash memory conversion address corresponding to the target fingerprint value;

[0027] Determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in the preset data table;

[0028] If it exists, obtain the disk ID corresponding to the target flash memory conversion address, obtain the target disk address according to the disk ID, and read the data on the target disk based on the target disk address.

[0029] In one embodiment, determining whether the data on the target disk is consistent with the target data includes:

[0030] Get the data on the target disk and the target data;

[0031] Parsing the data on the target disk and the target data into tree-structured data respectively;

[0032] According to the data in the tree structure, the data on the target disk and the target data are split respectively to obtain the target disk data block and the target data block;

[0033] Establishing target disk data blocks and mapping relationships between target data blocks according to the node types of the tree structure;

[0034] forming a plurality of mapping pairs based on the target disk data blocks and the semantic features corresponding to the target data blocks;

[0035] Compare the data on the target disk corresponding to the target disk data block and the target data corresponding to the target data block in the same mapping pair according to the character matching similarity and neighborhood structure information to obtain a comparison result;

[0036] Determine whether the data on the target disk is consistent with the target data based on the comparison results.

[0037] In one embodiment, in response to receiving a data deduplication instruction carrying information about data to be deleted, obtaining a logical address corresponding to the data to be deleted, and deleting the logical address corresponding to the data to be deleted from a preset data table includes:

[0038] Obtain the physical address corresponding to the data to be deleted, search the preset data table according to the physical address corresponding to the data to be deleted, and obtain the logical address linked list corresponding to the physical address corresponding to the data to be deleted;

[0039] The logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the logical address linked list corresponding to the physical address corresponding to the data to be deleted.

[0040] In order to solve the above technical problems, a second aspect provides a data processing device, comprising:

[0041] a calculation module, configured to calculate a target fingerprint value of the target data in response to receiving the target data sent by the host;

[0042] a processing module configured to obtain a preset data table and determine whether a fingerprint value matching the target fingerprint value exists in the preset data table; if not, write the target data and the target fingerprint value to disk based on a preset calculation rule; if so, obtain a flash translation layer address corresponding to the target fingerprint value and obtain a target disk address based on the flash translation layer address;

[0043] Read the data on the target disk based on the target disk address and determine whether the data on the target disk is consistent with the target data; if they are inconsistent, reallocate the target physical address to the target data and write the reallocated target physical address to the preset data table; if they are consistent, update the target logical address corresponding to the target data to the target logical address linked list corresponding to the target physical address in the preset data table;

[0044] The deletion module is used to, in response to receiving a data deduplication instruction carrying information of data to be deleted, obtain a logical address corresponding to the data to be deleted based on the information of the data to be deleted, and delete the logical address corresponding to the data to be deleted from a preset data table.

[0045] In order to solve the above technical problems, in the third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the following steps are implemented: when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0046] In order to solve the above technical problems, in a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0047] Different from the prior art, the present application sets a preset data table to determine whether there is a fingerprint value matching the target fingerprint value in the preset data table; if not, the target data and the target fingerprint value are written to disk based on a preset calculation rule; if so, the flash translation layer address corresponding to the target fingerprint value is obtained, and the target disk address is obtained based on the flash translation layer address; the data on the target disk is read based on the target disk address to determine whether the data on the target disk is consistent with the target data; if not, the target physical address is reallocated to the target data, and the reallocated target physical address is written to the preset data table; if consistent, the target logical address corresponding to the target data is updated to the target logical address linked list corresponding to the target physical address in the preset data table, and in response to receiving a data deduplication instruction carrying information about data to be deleted, the logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the preset data table. In the present application, by setting multiple judgment conditions, it is possible to accurately determine whether the target data is duplicate data and perform corresponding processing according to the judgment result, thereby improving system performance. In addition, in the present application, by deleting the logical address of the data to be deleted from the preset data table, the impact of deduplication on the processing capability of the software stack can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A diagram of an application environment of a data processing method in one embodiment;

[0049] Figure 2 is a flow chart of a data processing method in another embodiment;

[0050] Figure 3 is a structural block diagram of a data processing device in one embodiment;

[0051] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] Currently, to achieve better deduplication results, the industry's most commonly used deduplication method is software-level deduplication. However, enabling deduplication significantly impacts the entire software stack's ability to process I / O, causing a sharp decline in performance. This is primarily due to the increase in metadata overhead for the entire software stack caused by deduplication. This overhead includes CPU consumption and memory requirements. The CPU consumption primarily comes from the increased complexity of metadata calculations, while the memory requirement stems from the increased amount of metadata requiring caching.

[0054] To address these issues, the industry typically uses two solutions to mitigate performance degradation when deduplication is enabled. One solution involves calculating a hash value on the host data before writing it to disk to determine if it is duplicate data. If it is, the data does not need to be written to the disk device, saving storage space and reducing the number of writes. However, this solution has significant drawbacks: the deduplication function in the software stack increases the metadata overhead of the entire software stack, ultimately leading to a significant degradation in the software stack's I / O performance.

[0055] Another solution is to offload data processing from the software layer to the SSD disks (solid-state drives) that make up the thin pool. This has the benefit of reducing the software layer's CPU consumption and memory requirements for data processing, freeing up more resources for IO processing and improving performance. However, this solution also has obvious drawbacks. If data processing is completely offloaded to the disk without software adaptation, duplicate data can only be deduplicated within the disk, which does not match the array-level cross-disk deduplication requirements. As a result, the more disks in the array that make up the thin pool, the worse the deduplication effect.

[0056] In order to resolve the contradiction between software deduplication leading to low system performance and hard disk deduplication leading to poor deduplication effect, the present invention proposes a data processing method that can improve the deduplication ratio after offloading the deduplication function to the hard disk. This solution includes the cooperation of software and hardware levels. While ensuring the performance improvement brought by offloading the deduplication function to the hard disk, it also ensures the deduplication effect after offloading the deduplication function to the hardware.

[0057] In one embodiment, Figure 1 As shown, a data processing method is provided, which includes the following steps:

[0058] Step 101 : In response to receiving target data sent by a host, a target fingerprint value of the target data is calculated.

[0059] The target data sent by the host can be read-write data. After receiving the target data sent by the host, the target fingerprint value of the target data can be calculated using a hash algorithm. The hash algorithm maps a binary value of any length to a shorter binary value of a fixed length. This small binary value is called a hash value. The hash value can verify the integrity of the data, thereby improving the accuracy of data calculation.

[0060] Step 102: Obtain a preset data table and determine whether there is a fingerprint value matching the target fingerprint value in the preset data table.

[0061] Specifically, the preset data table may be a nested table that reflects the correspondence between the physical address and the logical address of the data storage. The preset data table may include a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores the physical address corresponding to the fingerprint data and a linked list of logical addresses pointing to each physical address.

[0062] For example, each disk device in this application is configured with multiple preset data tables. The multiple preset data tables for any disk device may include the preset data table corresponding to the data stored on that disk, as well as the preset data tables corresponding to the data stored on other disk devices in the system. The disk ID in the preset data table is used to quickly locate a specific disk device. Optionally, the disk device in this application may be an SSD (solid-state drive).

[0063] Fingerprint data includes the target data issued by the system and the target fingerprint value calculated based on the target data. The flash translation layer table represents the flash translation layer address corresponding to the target data. The flash translation layer address points to the physical address corresponding to the target data, i.e., the physical address where the target data is actually stored. Each physical address corresponds to a logical address linked list, which stores the logical addresses corresponding to that physical address. By adding the recording and maintenance of fingerprint data for other disk devices in the flash translation layer, array-level cross-disk deduplication can be achieved.

[0064] In this embodiment, after calculating the target fingerprint value of the target data sent by the current host, a search is performed from a preset data table to determine whether there is a fingerprint value identical to the target fingerprint value. If there is a fingerprint value identical to the target fingerprint value in the data table, it means that the target data sent by the current host may be duplicated with the data stored in the disk device.

[0065] Step 103: If the target data and the target fingerprint value do not exist, the target data and the target fingerprint value are written to disk based on a preset calculation rule.

[0066] In this embodiment, if there is no fingerprint value identical to the target fingerprint value in the preset data table, it means that the target data sent by the current host is new data, and the target data and the preset calculation rules of the target fingerprint value need to be stored and sent to a specific disk device for disk storage operation.

[0067] The operation of writing the target data and the target fingerprint value to the disk based on the preset calculation rules includes:

[0068] S1: Obtain target data and target fingerprint value.

[0069] S2: Calculate the ASCII value of the target data using the target fingerprint value as a seed, and assign a random number based on the ASCII value of the target data as the seed.

[0070] ASCII is a character encoding standard that assigns numbers to various characters. For example, target data A is assigned numbers in the range of 1-9, and target data AB is assigned numbers in the range of 10-99. The random number assigned to the target fingerprint value corresponding to the target data is based on the ASCII value of the target data.

[0071] S3: Obtain the number of disks constituting the array in the system.

[0072] S4: Using the number of disks constituting the array as the denominator and the random number as the numerator, take the remainder of the random number and the number of disks constituting the array to obtain the remainder.

[0073] S5: Search for the target data and the target disk corresponding to the target fingerprint value according to the remainder.

[0074] S6: Download the target data and target fingerprint value to the target disk.

[0075] Using the total number of disk devices in the system as the denominator and the random number corresponding to each target data as the numerator, the remainder of the random number corresponding to each data and the total number of disk devices in the system is calculated. The disk device corresponding to the remainder is used as the target disk for the target data, and the target data and the target fingerprint value are flushed to the disk. In this embodiment, the disk device corresponding to the remainder can be the disk device number, and the disk device with the same disk device number as the calculated remainder is selected as the target disk. By performing the flush operation for the target data in the above embodiment, wear balance can be ensured between the disk devices, which is conducive to extending the life of the disk devices.

[0076] Among them, step S6: flushing the target data and the target fingerprint value to the target disk includes: obtaining the target logical address and the target physical address corresponding to the target data; updating the target logical address and the target physical address to the flash memory translation layer table; in response to the target data and the target fingerprint value being flushed to the target disk, generating a successful disk write instruction, obtaining the target fingerprint value and the flash memory translation layer address corresponding to the target data, and inserting the target fingerprint value and the flash memory translation layer address corresponding to the target data into a preset data table.

[0077] In this application, data is written to the disk through the engine on the disk device, and the logical address and physical address of the data are allocated and updated to the flash translation layer table of the disk device. When the disk is successfully written, the fingerprint value and flash translation layer address corresponding to the data are notified to the software layer (the IO stack above the hard disk layer). The software layer updates the preset data table and inserts the received fingerprint value and flash translation layer address into the correct position in the preset data table. If the disk writing fails, the software layer is notified of the disk writing failure and there is no need to update the preset data table.

[0078] The fingerprint data and flash translation layer table in this application are stored in the form of key-value pairs, where the fingerprint data is the key and the flash translation layer table is the value. When the fingerprint value and the flash translation layer address are inserted into the correct position in the preset data table, certain rules are required to insert the fingerprint value and the flash translation layer address into the preset data table.

[0079] The certain rules here can be to sort the key values ​​from small to large or from large to small according to the numerical value of the fingerprint value in the fingerprint data. When the fingerprint value and the flash translation layer address are inserted into the preset data table, the order of the key values ​​of the fingerprint data to be inserted in the table is calculated, and the fingerprint data and the flash translation layer table are inserted into the correct position in the preset data table according to the order. The above implementation method can maintain an orderly preset data table to facilitate data management.

[0080] Step 104: If it exists, obtain the flash translation layer address corresponding to the target fingerprint value, and obtain the target disk address based on the flash translation layer address.

[0081] If it exists, obtain the flash memory conversion address corresponding to the target fingerprint value; determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in the preset data table; if it exists, obtain the disk ID corresponding to the target flash memory conversion address, obtain the target disk address according to the disk ID, and read the data on the target disk based on the target disk address.

[0082] That is, the flash translation layer address corresponding to the target fingerprint value is searched from the preset data table, and the disk ID of the preset data table with the same flash translation layer address as the flash translation layer address is searched from any disk according to the flash translation layer address, and the target disk position corresponding to the target data is located according to the disk ID.

[0083] Step 105 : Read the data on the target disk based on the target disk address, and determine whether the data on the target disk is consistent with the target data.

[0084] To further improve the accuracy of duplicate data identification, after finding a fingerprint value that matches the target fingerprint value of the target data sent by the current host in a preset data table, this application does not directly determine that the target data sent by the current host is duplicate data and directly perform a data deduplication operation. This application also provides a method for further obtaining data on the target disk to determine whether the data on the target disk is consistent with the target data.

[0085] Specifically, determining whether the data on the target disk is consistent with the target data includes:

[0086] Step 1051: Acquire the data on the target disk and the target data.

[0087] Step 1052: Parse the data on the target disk and the target data into tree-structured data respectively.

[0088] Step 1053: Split the data on the target disk and the target data according to the tree-structured data to obtain target disk data blocks and target data blocks.

[0089] Specifically, XML can be used to parse the target data and the data on the target disk into a tree structure. Specifically, the root node can be the source of the data (data on the target, target data), and the node can be the type of data (text, general characters, location information); the target data and the data on the target disk are parsed according to the nodes (data types) of the data in the tree structure to obtain the target disk data block and the target data block respectively. The target data block includes: target data text, target data general characters, target data location information; the target disk data block includes: data text on the target disk, data general characters on the target disk, and data location information on the target disk.

[0090] Step 1054: Establish target disk data blocks and mapping relationships between target data blocks according to the node types of the tree structure.

[0091] Step 1055: forming a plurality of mapping pairs based on the target disk data blocks and the semantic features corresponding to the target data blocks.

[0092] A mapping relationship is established between the target disk data blocks and the target data blocks of the same data type. The target data blocks and the target disk data blocks are further divided based on semantic information. Data pairs corresponding to the target data blocks and the target disk data blocks are obtained. For example, the target data body and the target disk data body are described as an example. Further division can be performed based on the semantic information of the body. Assuming that the semantic information of the target data block body includes device A, which is used to store device information and controls device B to perform corresponding operations based on the storage information, and the semantic information of the target disk data block includes device B, which is used to control device A to perform corresponding operations, analysis is performed based on the semantic information represented by A and B. For example, if device A is a storage and control device and device B is a control device, the relevant information of the control part of device A is extracted and associated with the relevant information of the control part of device B to obtain data pairs corresponding to the target data blocks and the target disk data blocks. A mapping relationship is established between each data pair to obtain a mapping pair.

[0093] Step 1056: Compare the data on the target disk corresponding to the target disk data block in the same mapping pair and the target data corresponding to the target data block according to the character matching similarity and neighborhood structure information to obtain a comparison result.

[0094] Step 1057: Determine whether the data on the target disk is consistent with the target data based on the comparison result.

[0095] Obtain nodes in the data structure that have not been matched into data pairs and count the number of matched nodes around these nodes. Here, "surrounding" refers to adjacent nodes of the same node level that have not been matched into data pairs. From the unmatched nodes in the baseline directory tree, select the node with the most matching surrounding nodes. Calculate the similarity between this node with the most matching surrounding nodes and the node that has not been matched into a data pair. Based on this similarity, determine whether the data on the target disk is consistent with the target data.

[0096] Step 106: If there is no consistency, reallocate the target physical address for the target data, and write the reallocated target physical address into a preset data table.

[0097] If the data on the target disk is inconsistent with the target data sent by the host, the target data sent by the host is considered non-duplicate data, and a target physical address needs to be assigned to the target data, or the target data needs to be forwarded to another disk for disk processing. After the target data has been assigned a target physical address or forwarded to another disk for disk processing, the target physical address and target logical address corresponding to the target data are obtained and inserted into the flash translation layer table corresponding to the target data.

[0098] Step 107: If they are consistent, the target logical address corresponding to the target data is updated into the target logical address linked list corresponding to the target physical address in the preset data table.

[0099] When the data on the target disk is consistent with the target data sent by the host, the target data sent by the host is considered to be duplicate data, and the target logical address corresponding to the target data needs to be updated to the target logical address linked list corresponding to the target physical address in the preset data table. Specifically, the target physical address corresponding to the target data is obtained, the preset data table is searched according to the target physical address, the target logical address corresponding to the target data is obtained from the standard position of the target physical address in the preset data table, the target logical address is inserted into the target logical address linked list at the standard position, and the preset data table is updated.

[0100] A physical address in this application can be a target physical address corresponding to multiple target numbers, and each target physical address can correspond to multiple target logical addresses. When it is detected that the target data is duplicate data, it is only necessary to update the target logical address corresponding to the target data to the logical address linked list under the target object address corresponding to the target data in the flash memory conversion layer of the disk.

[0101] For example, assume that the target logical address LBA1 points to the target physical address PBA1, the target logical address LBA2 points to the target physical address PBA2, and the data corresponding to the target logical address LBA2 is consistent with the data corresponding to the target logical address LBA1. In the case of deduplication, the target logical address LBA2 also points to the target physical address PBA1. Originally, only the target logical address LBA1 corresponds to the target physical address PBA1. In this case, the target logical address LBA2 needs to be added to the logical address (LBA) linked list of the target physical address PBA1. This means that there are two logical addresses (LBAs) pointing to the target physical address PBA1.

[0102] Step 108 : In response to receiving the data deduplication instruction carrying the data to be deleted information, obtain the logical address corresponding to the data to be deleted based on the data to be deleted information, and delete the logical address corresponding to the data to be deleted from the preset data table.

[0103] In the prior art, the host writes IO at a certain granularity. Assuming that each IO written is called a data block, under normal data deduplication, a basically unique fingerprint value can be calculated for each data block. During data deduplication, fingerprint value A of data block A is calculated, and then this fingerprint value A is used to check the mapping relationship between all fingerprint values ​​and PBAs. If no fingerprint value is found, a new PBA is allocated. If a PBA is found, no new PBA needs to be allocated and the found PBA is directly used. In this way, data block A is equivalent to being deduplicated because no new PBA is allocated. This will have a significant impact on the ability of the entire software stack to process IO.

[0104] The present invention proposes a solution that can improve the deduplication ratio after offloading the deduplication function to the disk device. The solution includes cooperation at the software and hardware levels. While ensuring the performance improvement brought by offloading the deduplication function to the disk device, it also ensures the deduplication effect after offloading the deduplication to the hardware.

[0105] When a deduplication instruction is received, the system obtains the data to be deleted information carried in the instruction. The data to be deleted information includes the physical address corresponding to the data to be deleted, the logical address corresponding to the data to be deleted, and a list of logical addresses corresponding to the data to be deleted. Therefore, when the deduplication instruction is executed, the system only needs to delete the logical address corresponding to the data to be deleted from the preset data table.

[0106] In the present application, by storing a preset data table in a disk device, a disk in the present application stores a preset data table corresponding to the data stored in the disk and other disks. It is only necessary to delete the logical address corresponding to the data to be deleted from the preset data table. In this way, there is no correspondence between the logical address corresponding to the data to be deleted and the physical address of the data to be deleted in the preset data table, which is equivalent to deleting the data to be deleted.

[0107] In a feasible implementation, the present application also includes obtaining the system's performance indicators during a preset time period, setting weight values ​​for the system's performance indicators, calculating the system's average load based on the system's performance indicators and the weight values ​​corresponding to the system's performance indicators, and comparing the calculated system's average load with the preset system's maximum load; and adjusting the proportion of background deduplication processing diversion based on the comparison results.

[0108] The preset time period can be to obtain the system performance indicators once every certain period of time. Performance indicators may include CPU usage, disk device usage, memory usage, etc. Weight values ​​can be set for the system performance indicators based on actual experience. For example, in a scenario with high CPU requirements, the weight of the CPU usage can be set higher than the weights of other performance indicators. This application does not limit the specific values ​​of the weights. The average load of the system can be calculated using the system average load load average calculation formula, where the system average load load average calculation formula is as follows:

[0109] load1=load0×e+active×w(1-e)

[0110] Where load1 represents the system load at time t+1, load0 represents the system load at time t, e represents the attenuation coefficient, active represents the number of active processes in the system, and w represents the product of the set weights.

[0111] Compare the calculated system average load with the preset system maximum load. Assume that all system write IO requests are deduplicated in the background. That is, the system load when the background deduplication processing data diversion ratio is 100% is the system maximum load. Compare the system average load with the system maximum load.

[0112] Based on the comparison results, the data split between background deduplication and online deduplication is adjusted. If the system average load is not less than the maximum system load, the data split between background deduplication and online deduplication is set to 100%, indicating that all data is split between background and online deduplication. If the system average load is less than the maximum system load, the system average load is compared with the average system load calculated at the previous moment. If the system average load is greater than the average system load calculated at the previous moment, the data split between background and online deduplication is appropriately increased. If the system average load is less than the average system load calculated at the previous moment, the data split between background and online deduplication is appropriately reduced.

[0113] By regularly sampling and statistically analyzing CPU, memory, and disk usage during storage system operation, and combining different weights to estimate the actual system load, we dynamically adjust the background deduplication ratio based on the system load. This hardware-resource-based estimation method can more accurately reflect the current load, reduce I / O response latency, and improve system performance.

[0114] See also Figure 2 In one embodiment, in order to better illustrate the data processing method provided by the present application, the data processing method of the present application is described with a specific example:

[0115] (1) The software layer (IO stack above the hard disk layer) maintains an SSD FP Table (preset data table) to manage the fingerprint data of the entire thin pool and the FTL address (Flash Translation Layer address) of the disk corresponding to the fingerprint data. The table granularity (preset data table strength) matches the current system data block granularity.

[0116] The table granularity (preset data table) here matches the current system data block granularity, which means that a preset data table stores the fingerprint data of a data block and the FTL address (flash translation layer address) corresponding to the fingerprint data.

[0117] (2) The IO (target data) sent by the host passes through the software stack, calculates the corresponding fingerprint value (target fingerprint value), and retrieves the fingerprint value in the SSD FP Table (preset data table).

[0118] (3) If in step (2), no fingerprint value that is consistent with the target fingerprint value of the target data newly sent by the host is retrieved from the SSD FP Table (preset data table), it means that the data is the first data. It is necessary to calculate the specific SSD disk to which the IO data (target data) and the calculated fingerprint value need to be sent according to certain calculation rules and send them to the SSD disk together.

[0119] (4) The data sent to the SSD is written to the disk through the engine on the disk, and the LBA address (logical address) and the assigned PBA address (physical address) of the data are updated to the FTL table (Flash Translation Layer Table) of the disk. If the disk is successfully written, the fingerprint value and FTL address (Flash Translation Layer address) are notified to the software layer. The software layer updates the SSD FP Table (preset data table) and inserts the received fingerprint value and FTL address (Flash Translation Layer address) into the correct position of the SSD FP Table (preset data table). If the disk is written to the disk, the software layer is notified of the failure and there is no need to update the SSD FP Table (preset data table).

[0120] (5) If in step (2), a fingerprint value that is consistent with the fingerprint value of the sent data is retrieved from the SSD FP Table (preset data table), it means that it may be duplicate data, and the FTL address (flash translation layer address) corresponding to the fingerprint value is obtained.

[0121] (6) Find the corresponding location of the FTL table (Flash Translation Layer table) of the SSD disk based on the FTL address (Flash Translation Layer address), and obtain the PBA address (Physical Address) stored therein, read the data from the disk based on the PBA address (Physical Address), and compare it byte by byte with the new data sent by the host.

[0122] (7) In step (6), if the two copies of data are found to be inconsistent after comparison, it means that they are not duplicate data. A new PBA address (physical address) needs to be assigned to the data, or it needs to be forwarded to another disk for storage. The LBA address (logical address) and PBA address (physical address) of the newly sent data are inserted into the appropriate position of the corresponding FTL table (flash translation layer table), and the storage result is returned to the software layer.

[0123] (8) If the two copies of data are consistent after byte-by-byte comparison in step (6), it means that they are indeed duplicate data. You only need to update the new LBA address (logical address) to the LBA address linked list (logical address linked list) corresponding to the original PBA address (physical address) in the FTL table (Flash Translation Layer table) of the disk, and notify the software layer that the update is successful.

[0124] (9) When the SSD disk performs garbage collection subsequently, it is necessary to refer to the LBA address list (logical address list) corresponding to each PBA (physical address) in the FTL table (flash translation layer table). Only when the LBA address list (logical address list) is empty can the PBA (physical address) be recycled.

[0125] In the existing technology, it is usually only necessary to maintain the one-to-one mapping relationship between the LBA (logical address) and PBA (physical address) corresponding to the data. However, for deleted data, multiple LBAs (logical addresses) point to the same PBA (physical address). Therefore, there needs to be a way to know the number of LBAs (logical addresses) corresponding to the current PBA (physical address) to facilitate capacity statistics and garbage collection. Only when the current PBA (physical address) is not referenced by any LBA (logical address) can the invalid PBA (physical address) be recycled. The statement here describes how to implement the mapping relationship between PBA (physical address) and LBA (logical address).

[0126] LBA (logical address) is allocated according to volume space. Volume space is fixed, so the LBA address (logical address) corresponding to each volume is fixed. Suppose when LBA1 is written for the first time, the physical address assigned is PBA1. When LBA2 is written, it is the same data as LBA1, that is, duplicate data. At this time, there is no need to allocate a new PBA, and let it correspond to PBA1. At this time, there are two LBA addresses corresponding to PBA1. When LBA1 is overwritten, it may be new data. At this time, a new PBA2 needs to be allocated for LBA1, and LBA1 needs to be removed from the LBA linked list of PBA1. Similarly, if LBA2 is overwritten with new data, a new physical address PBA3 is allocated. In this case, the LBA linked list corresponding to PBA1 is empty, and the PBA (physical address) corresponding to the LBA table (logical address linked list) needs to be recovered.

[0127] This application adds the recording and maintenance of fingerprint data of other disk devices to the preset data table of the disk device, thereby solving the problem that after the deduplication function is offloaded to the hard disk, only deduplication can be performed within the disk, resulting in the more disks in the array that constitute the thin pool, the worse the deduplication data removal effect. This allows the entire storage system to obtain both the performance improvement brought by offloading deduplication to the hard disk and a better deduplication effect.

[0128] It should be understood that although Figure 1-Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0129] In one embodiment, Figure 3 As shown, a data processing device is provided, including: a calculation module 30, a processing module 31 and a deletion module 32, wherein:

[0130] The calculation module 30 is configured to calculate a target fingerprint value of the target data in response to receiving the target data sent by the host;

[0131] Processing module 31 is configured to obtain a preset data table and determine whether a fingerprint value matching the target fingerprint value exists in the preset data table; if not, write the target data and the target fingerprint value to disk based on a preset calculation rule; if so, obtain the flash translation layer address corresponding to the target fingerprint value and obtain the target disk address based on the flash translation layer address;

[0132] Read the data on the target disk based on the target disk address and determine whether the data on the target disk is consistent with the target data; if they are inconsistent, reallocate the target physical address to the target data and write the reallocated target physical address to the preset data table; if they are consistent, update the target logical address corresponding to the target data to the target logical address linked list corresponding to the target physical address in the preset data table;

[0133] The deletion module is used to respond to receiving a data deduplication instruction carrying information of data to be deleted, obtain the logical address corresponding to the data to be deleted based on the information of the data to be deleted, and delete the logical address corresponding to the data to be deleted from a preset data table.

[0134] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0135] The preset data table includes a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores physical addresses corresponding to the fingerprint data and a logical address linked list formed by logical addresses pointing to each physical address.

[0136] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0137] The operations of writing target data and target fingerprint values ​​to disk based on preset calculation rules include:

[0138] Obtain target data and target fingerprint value;

[0139] Using the target fingerprint value as a seed, calculate the ASCII value of the target data, and assign a random number to the seed based on the ASCII value of the target data;

[0140] Get the number of disks that make up the array in the system;

[0141] The remainder is obtained by taking the number of disks constituting the array as the denominator and the random number as the numerator and multiplying the random number by the number of disks constituting the array.

[0142] Find the target data and the target disk corresponding to the target fingerprint value according to the remainder;

[0143] Download the target data and target fingerprint value to the target disk.

[0144] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0145] Flashing the target data and target fingerprint value to the target disk also includes:

[0146] Obtain the target logical address and target physical address corresponding to the target data;

[0147] Update the target logical address and the target physically into the flash translation layer table;

[0148] In response to the target data and the target fingerprint value being flushed to the target disk, a successful flush instruction is generated, the target fingerprint value and the flash translation layer address corresponding to the target data are obtained, and the target fingerprint value and the flash translation layer address corresponding to the target data are inserted into a preset data table.

[0149] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0150] Obtaining a flash translation layer address corresponding to a target fingerprint value, and obtaining a target disk address based on the flash translation layer address includes:

[0151] Get the flash memory conversion address corresponding to the target fingerprint value;

[0152] Determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in the preset data table;

[0153] If it exists, obtain the disk ID corresponding to the target flash memory conversion address, obtain the target disk address according to the disk ID, and read the data on the target disk based on the target disk address.

[0154] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0155] Determining whether the data on the target disk is consistent with the target data includes:

[0156] Get the data on the target disk and the target data;

[0157] Parsing the data on the target disk and the target data into tree-structured data respectively;

[0158] According to the data in the tree structure, the data on the target disk and the target data are split respectively to obtain the target disk data block and the target data block;

[0159] Establishing target disk data blocks and mapping relationships between target data blocks according to the node types of the tree structure;

[0160] forming a plurality of mapping pairs based on the target disk data blocks and the semantic features corresponding to the target data blocks;

[0161] Compare the data on the target disk corresponding to the target disk data block and the target data corresponding to the target data block in the same mapping pair according to the character matching similarity and neighborhood structure information to obtain a comparison result;

[0162] Determine whether the data on the target disk is consistent with the target data based on the comparison results.

[0163] In one embodiment, the above device can implement another implementation of a data processing method, and the specific steps are as follows:

[0164] Acquiring a logical address corresponding to the data to be deleted based on the information of the data to be deleted, and deleting the logical address corresponding to the data to be deleted from a preset data table includes:

[0165] Obtain the physical address corresponding to the data to be deleted, search the preset data table according to the physical address corresponding to the data to be deleted, and obtain the logical address linked list corresponding to the physical address corresponding to the data to be deleted;

[0166] The logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the logical address linked list corresponding to the physical address corresponding to the data to be deleted.

[0167] For the specific definition of the data processing device, please refer to the definition of the data processing method above and will not be repeated here. Each module in the above-mentioned data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0168] In one embodiment, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data processing methods provided by the above methods.

[0169] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data applied by the data processing method in this application. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0170] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0171] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0172] Step 101: In response to receiving target data sent by a host, a target fingerprint value of the target data is calculated.

[0173] Step 102: Obtain a preset data table and determine whether there is a fingerprint value matching the target fingerprint value in the preset data table.

[0174] Step 103: If the target data and the target fingerprint value do not exist, the target data and the target fingerprint value are written to disk based on a preset calculation rule.

[0175] Step 104: If it exists, obtain the flash translation layer address corresponding to the target fingerprint value, and obtain the target disk address based on the flash translation layer address.

[0176] Step 105 : Read the data on the target disk based on the target disk address, and determine whether the data on the target disk is consistent with the target data.

[0177] Step 106: If there is no consistency, reallocate the target physical address for the target data, and write the reallocated target physical address into a preset data table.

[0178] Step 107: If they are consistent, the target logical address corresponding to the target data is updated into the target logical address linked list corresponding to the target physical address in the preset data table.

[0179] Step 108: In response to receiving the data deduplication instruction carrying the data to be deleted information, obtain the logical address corresponding to the data to be deleted based on the data to be deleted information, and delete the logical address corresponding to the data to be deleted from the preset data table.

[0180] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0181] The preset data table includes a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores physical addresses corresponding to the fingerprint data and a logical address linked list formed by logical addresses pointing to each physical address.

[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0183] The operations of writing target data and target fingerprint values ​​to disk based on preset calculation rules include:

[0184] Obtain target data and target fingerprint value;

[0185] Using the target fingerprint value as a seed, calculate the ASCII value of the target data, and assign a random number to the seed based on the ASCII value of the target data;

[0186] Get the number of disks that make up the array in the system;

[0187] The remainder is obtained by taking the number of disks constituting the array as the denominator and the random number as the numerator and multiplying the random number by the number of disks constituting the array.

[0188] Find the target data and the target disk corresponding to the target fingerprint value according to the remainder;

[0189] Download the target data and target fingerprint value to the target disk.

[0190] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0191] Flashing the target data and target fingerprint value to the target disk also includes:

[0192] Obtain the target logical address and target physical address corresponding to the target data;

[0193] Update the target logical address and the target physically into the flash translation layer table;

[0194] In response to the target data and the target fingerprint value being flushed to the target disk, a successful flush instruction is generated, the target fingerprint value and the flash translation layer address corresponding to the target data are obtained, and the target fingerprint value and the flash translation layer address corresponding to the target data are inserted into a preset data table.

[0195] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0196] Obtaining a flash translation layer address corresponding to a target fingerprint value, and obtaining a target disk address based on the flash translation layer address includes:

[0197] Get the flash memory conversion address corresponding to the target fingerprint value;

[0198] Determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in the preset data table;

[0199] If it exists, obtain the disk ID corresponding to the target flash memory conversion address, obtain the target disk address according to the disk ID, and read the data on the target disk based on the target disk address.

[0200] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0201] Determining whether the data on the target disk is consistent with the target data includes:

[0202] Get the data on the target disk and the target data;

[0203] Parsing the data on the target disk and the target data into tree-structured data respectively;

[0204] According to the data in the tree structure, the data on the target disk and the target data are split respectively to obtain the target disk data block and the target data block;

[0205] Establishing target disk data blocks and mapping relationships between target data blocks according to the node types of the tree structure;

[0206] forming a plurality of mapping pairs based on the target disk data blocks and the semantic features corresponding to the target data blocks;

[0207] Compare the data on the target disk corresponding to the target disk data block and the target data corresponding to the target data block in the same mapping pair according to the character matching similarity and neighborhood structure information to obtain a comparison result;

[0208] Determine whether the data on the target disk is consistent with the target data based on the comparison results.

[0209] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0210] Acquiring a logical address corresponding to the data to be deleted based on the information of the data to be deleted, and deleting the logical address corresponding to the data to be deleted from a preset data table includes:

[0211] Obtain the physical address corresponding to the data to be deleted, search the preset data table according to the physical address corresponding to the data to be deleted, and obtain the logical address linked list corresponding to the physical address corresponding to the data to be deleted;

[0212] The logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the logical address linked list corresponding to the physical address corresponding to the data to be deleted.

[0213] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0214] Step 101: In response to receiving target data sent by a host, a target fingerprint value of the target data is calculated.

[0215] Step 102: Obtain a preset data table and determine whether there is a fingerprint value matching the target fingerprint value in the preset data table.

[0216] Step 103: If the target data and the target fingerprint value do not exist, the target data and the target fingerprint value are written to disk based on a preset calculation rule.

[0217] Step 104: If it exists, obtain the flash translation layer address corresponding to the target fingerprint value, and obtain the target disk address based on the flash translation layer address.

[0218] Step 105 : Read the data on the target disk based on the target disk address, and determine whether the data on the target disk is consistent with the target data.

[0219] Step 106: If there is no consistency, reallocate the target physical address for the target data, and write the reallocated target physical address into a preset data table.

[0220] Step 107: If they are consistent, the target logical address corresponding to the target data is updated into the target logical address linked list corresponding to the target physical address in the preset data table.

[0221] Step 108: In response to receiving the data deduplication instruction carrying the data to be deleted information, obtain the logical address corresponding to the data to be deleted based on the data to be deleted information, and delete the logical address corresponding to the data to be deleted from the preset data table.

[0222] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0223] The preset data table includes a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores physical addresses corresponding to the fingerprint data and a logical address linked list formed by logical addresses pointing to each physical address.

[0224] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0225] The operations of writing target data and target fingerprint values ​​to disk based on preset calculation rules include:

[0226] Obtain target data and target fingerprint value;

[0227] Using the target fingerprint value as a seed, calculate the ASCII value of the target data, and assign a random number to the seed based on the ASCII value of the target data;

[0228] Get the number of disks that make up the array in the system;

[0229] The remainder is obtained by taking the number of disks constituting the array as the denominator and the random number as the numerator and multiplying the random number by the number of disks constituting the array.

[0230] Find the target data and the target disk corresponding to the target fingerprint value according to the remainder;

[0231] Download the target data and target fingerprint value to the target disk.

[0232] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0233] Flashing the target data and target fingerprint value to the target disk also includes:

[0234] Obtain the target logical address and target physical address corresponding to the target data;

[0235] Update the target logical address and the target physically into the flash translation layer table;

[0236] In response to the target data and the target fingerprint value being flushed to the target disk, a successful flush instruction is generated, the target fingerprint value and the flash translation layer address corresponding to the target data are obtained, and the target fingerprint value and the flash translation layer address corresponding to the target data are inserted into a preset data table.

[0237] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0238] Obtaining a flash translation layer address corresponding to a target fingerprint value, and obtaining a target disk address based on the flash translation layer address includes:

[0239] Get the flash memory conversion address corresponding to the target fingerprint value;

[0240] Determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in the preset data table;

[0241] If it exists, obtain the disk ID corresponding to the target flash memory conversion address, obtain the target disk address according to the disk ID, and read the data on the target disk based on the target disk address.

[0242] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0243] Determining whether the data on the target disk is consistent with the target data includes:

[0244] Get the data on the target disk and the target data;

[0245] Parsing the data on the target disk and the target data into tree-structured data respectively;

[0246] According to the data in the tree structure, the data on the target disk and the target data are split respectively to obtain the target disk data block and the target data block;

[0247] Establishing target disk data blocks and mapping relationships between target data blocks according to the node types of the tree structure;

[0248] forming a plurality of mapping pairs based on the target disk data blocks and the semantic features corresponding to the target data blocks;

[0249] Compare the data on the target disk corresponding to the target disk data block and the target data corresponding to the target data block in the same mapping pair according to the character matching similarity and neighborhood structure information to obtain a comparison result;

[0250] Determine whether the data on the target disk is consistent with the target data based on the comparison results.

[0251] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0252] Acquiring a logical address corresponding to the data to be deleted based on the information of the data to be deleted, and deleting the logical address corresponding to the data to be deleted from a preset data table includes:

[0253] Obtain the physical address corresponding to the data to be deleted, search the preset data table according to the physical address corresponding to the data to be deleted, and obtain the logical address linked list corresponding to the physical address corresponding to the data to be deleted;

[0254] The logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the logical address linked list corresponding to the physical address corresponding to the data to be deleted.

[0255] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0256] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0257] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: include: In response to receiving the target data sent by the host, calculating the target fingerprint value of the target data; Obtain a preset data table, and determine whether there is a fingerprint value matching the target fingerprint value in the preset data table; If not, performing a write operation on the target data and the target fingerprint value based on a preset calculation rule; If it exists, obtaining the flash translation layer address corresponding to the target fingerprint value, and obtaining the target disk address based on the flash translation layer address; Reading data on the target disk based on the target disk address, and determining whether the data on the target disk is consistent with the target data; If they are inconsistent, reallocate the target physical address for the target data and write the reallocated target physical address into the preset data table; If they are consistent, the target logical address corresponding to the target data is updated to the target logical address linked list corresponding to the target physical address in the preset data table; In response to receiving a data deduplication instruction carrying information of data to be deleted, a logical address corresponding to the data to be deleted is obtained based on the information of the data to be deleted, and the logical address corresponding to the data to be deleted is deleted from a preset data table.

2. The method according to claim 1, characterized in that The preset data table includes a disk ID, fingerprint data, and a flash translation layer table. The flash translation layer table stores a physical address corresponding to the fingerprint data and a logical address linked list formed by a logical address pointing to each physical address.

3. The method according to claim 1, characterized in that The performing the disk operation on the target data and the target fingerprint value based on the preset calculation rule includes: Obtain target data and target fingerprint value; Using the target fingerprint value as a seed, calculating the ASCII value of the target data, and assigning a random number to the seed based on the ASCII value of the target data; Get the number of disks that make up the array in the system; Taking the number of disks constituting the array as the denominator and the random number as the numerator, taking the remainder of the random number and the number of disks constituting the array to obtain a remainder; Searching for the target data and the target disk corresponding to the target fingerprint value according to the remainder; The target data and the target fingerprint value are downloaded to the target disk.

4. The method according to claim 2, characterized in that The step of flashing the target data and the target fingerprint value to the target disk further includes: Obtain the target logical address and target physical address corresponding to the target data; physically updating the target logical address and the target into a flash translation layer table; In response to the target data and the target fingerprint value being flushed to the target disk, a successful flush instruction is generated, the target fingerprint value and the flash translation layer address corresponding to the target data are obtained, and the target fingerprint value and the flash translation layer address corresponding to the target data are inserted into a preset data table.

5. The method according to claim 1, wherein The obtaining of a flash translation layer address corresponding to a target fingerprint value, and obtaining the target disk address based on the flash translation layer address includes: Get the flash memory conversion address corresponding to the target fingerprint value; Determine whether there is a target flash memory conversion address matching the flash memory conversion address corresponding to the target fingerprint value in a preset data table; If so, obtain a disk ID corresponding to the target flash memory conversion address, obtain a target disk address according to the disk ID, and read data on the target disk based on the target disk address.

6. The method according to claim 1, characterized in that Determining whether the data on the target disk is consistent with the target data includes: Get the data on the target disk and the target data; Respectively parsing the data on the target disk and the target data into tree-structured data; According to the data in the tree structure, the data on the target disk and the target data are split respectively to obtain target disk data blocks and target data blocks; Establishing a mapping relationship between the target disk data block and the target data block according to the node type of the tree structure; forming a plurality of mapping pairs based on the semantic features corresponding to the target disk data block and the target data block; Compare the data on the target disk corresponding to the target disk data block and the target data corresponding to the target data block in the same mapping pair according to the character matching similarity and neighborhood structure information to obtain a comparison result; It is determined whether the data on the target disk is consistent with the target data according to the comparison result.

7. The method according to claim 1, characterized in that The acquiring the logical address corresponding to the data to be deleted based on the information of the data to be deleted, and deleting the logical address corresponding to the data to be deleted from the preset data table includes: Obtaining the physical address corresponding to the data to be deleted, searching a preset data table according to the physical address corresponding to the data to be deleted, and obtaining a logical address linked list corresponding to the physical address corresponding to the data to be deleted; The logical address corresponding to the data to be deleted is obtained, and the logical address corresponding to the data to be deleted is deleted from the logical address linked list corresponding to the physical address corresponding to the data to be deleted.

8. A data processing device, characterized in that: The device comprises: The calculation module is configured to calculate a target fingerprint value of the target data in response to receiving target data sent by the host; the processing module is configured to obtain a preset data table and determine whether a fingerprint value matching the target fingerprint value exists in the preset data table; if not, perform a disk write operation on the target data and the target fingerprint value based on a preset calculation rule; if so, obtain a flash translation layer address corresponding to the target fingerprint value, and obtain a target disk address based on the flash translation layer address; Reading data on the target disk based on the target disk address, determining whether the data on the target disk is consistent with the target data; if not, reallocating a target physical address for the target data and writing the reallocated target physical address into a preset data table; if consistent, updating the target logical address corresponding to the target data into a target logical address linked list corresponding to the target physical address in the preset data table; The deletion module is used to, in response to receiving a data deduplication instruction carrying information of data to be deleted, obtain a logical address corresponding to the data to be deleted based on the information of the data to be deleted, and delete the logical address corresponding to the data to be deleted from a preset data table.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images