Data processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data processing method, device and equipment and a computer readable storage medium, which are applied to the field of data processing, and the method comprises the following steps: obtaining a deduplication granularity interval; calculating a deduplication ratio corresponding to the deduplication granularity in the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval; and determining the optimal deduplication granularity from the optimal deduplication granularity interval, and deleting the duplicated data in the storage space according to the optimal deduplication granularity. According to the method, the deduplication granularity interval is adaptively adjusted through the deduplication ratio to obtain the optimal deduplication granularity interval, the optimal deduplication granularity is further determined from the optimal deduplication granularity interval, and data deduplication is carried out through the optimal deduplication granularity. According to the flexible and accurate deduplication granularity determination mode, normal operation of data deduplication can be ensured, the deduplication ratio can be remarkably increased, the storage space is saved, and the deduplication performance is further optimized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In actual applications, deduplication technology usually adopts a fixed granularity setting strategy. This single fixed granularity setting method often cannot achieve the best deduplication efficiency, resulting in a large amount of disk space occupied due to the large amount of data stored on the disk.

[0003] Therefore, how to improve the data deduplication efficiency of storage space is an urgent problem that needs to be solved. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a data processing method, apparatus, device and computer-readable storage medium, which solve the problem of low data deduplication efficiency in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a data processing method, comprising:

[0006] Get the deduplication granularity range;

[0007] Calculating a deduplication ratio corresponding to a deduplication granularity within the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval;

[0008] An optimal deduplication granularity is determined from the optimal deduplication granularity interval, and duplicate data in the storage space is deleted according to the optimal deduplication granularity.

[0009] On the one hand, obtain the deduplication granularity range, including:

[0010] Determine a minimum deduplication granularity and a maximum deduplication granularity according to the current write granularity; the minimum deduplication granularity is smaller than the current write granularity;

[0011] The deduplication granularity interval is determined according to the minimum deduplication granularity and the maximum deduplication granularity.

[0012] On the one hand, it also includes:

[0013] Acquire a data fingerprint of the data written into the storage space, and cache the data fingerprint into the storage space; the data fingerprint is set according to the data segment structure;

[0014] The remaining data after deletion from the storage space is written into the hard disk according to the data fingerprint and metadata management system.

[0015] On the one hand, writing the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system includes:

[0016] When the hard disk is empty, all the remaining data are written into the hard disk according to the data fingerprint;

[0017] When the hard disk is not empty, the remaining data is compared with the data in the hard disk for similarity according to the data fingerprint to determine the same data and different data in the remaining data;

[0018] The different data are written into the hard disk.

[0019] On the one hand, writing the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system includes:

[0020] Obtaining the number of write times, and dividing the remaining data into hot data and cold data according to the number of write times;

[0021] The hot data is written to the center of the disk according to the data fingerprint, and the cold data is written to the edge of the disk according to the data fingerprint.

[0022] On the one hand, calculating the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval, including:

[0023] Calculating deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity, and the middle deduplication granularity in the deduplication granularity interval respectively;

[0024] updating the deduplication granularity interval according to the deduplication ratio;

[0025] Repeat the steps of respectively calculating the deduplication ratios at the minimum deduplication granularity, the maximum deduplication granularity, and the intermediate deduplication granularity in the deduplication granularity interval and updating the deduplication granularity interval according to the deduplication ratios until the difference between the maximum deduplication granularity and the minimum deduplication granularity of the updated deduplication granularity interval meets a preset threshold, stop updating, and obtain the optimal deduplication granularity interval;

[0026] Wherein, updating the deduplication granularity interval according to the deduplication ratio includes:

[0027] The deduplication granularity corresponding to the calculated maximum deduplication ratio is used as the maximum deduplication granularity of the deduplication granularity interval, and the deduplication granularity corresponding to the calculated second largest deduplication ratio is used as the minimum deduplication granularity of the deduplication granularity interval.

[0028] On the one hand, determining the optimal deduplication granularity from the optimal deduplication granularity interval includes:

[0029] The middle value of the optimal deduplication granularity interval is used as the optimal deduplication granularity.

[0030] The present invention also provides a data processing device, comprising:

[0031] A deduplication ratio interval acquisition module is used to obtain a deduplication granularity interval;

[0032] an optimal deduplication granularity interval determination module, configured to calculate a deduplication ratio corresponding to a deduplication granularity within the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval;

[0033] The deduplication module is used to determine the optimal deduplication granularity from the optimal deduplication granularity interval, and delete duplicate data in the storage space according to the optimal deduplication granularity.

[0034] The present invention also provides a data processing device, comprising:

[0035] Memory for storing computer programs;

[0036] A processor is used to implement the steps of the above-mentioned data processing method when executing the computer program.

[0037] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the steps of the data processing method as described above are implemented.

[0038] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above data processing method when executed by a processor.

[0039] It can be seen from the above technical scheme that the present invention obtains the deduplication granularity interval; calculates the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval, and adjusts the deduplication granularity interval according to the deduplication ratio to obtain the optimal deduplication granularity interval; determines the optimal deduplication granularity from the optimal deduplication granularity interval, and deletes duplicate data in the storage space according to the optimal deduplication granularity. The beneficial effect of the present invention is: the optimal deduplication granularity interval is obtained by adaptively adjusting the deduplication granularity interval through the deduplication ratio, and then the optimal deduplication granularity is further determined from the optimal deduplication granularity interval, and data deduplication is performed using the optimal deduplication granularity. Through this flexible and accurate deduplication granularity determination method, not only can the normal operation of deduplication be ensured, but also the deduplication ratio can be significantly improved, storage space can be saved, and deduplication performance can be further optimized.

[0040] In addition, the present invention also provides a data processing device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0043] Figure 2 An example diagram of an iterative optimization process provided by an embodiment of the present invention;

[0044] Figure 3 A flowchart of a data processing method provided by an embodiment of the present invention;

[0045] Figure 4 An example diagram of a process of storing data on a disk provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0047] Figure 6 A structural schematic diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] First, some terms used in this application are explained:

[0050] Deduplication granularity: refers to the minimum unit size used to identify and compare data blocks during the deduplication process.

[0051] Data fingerprint: A fixed-length, unique numerical value or string generated by processing raw data through a specific algorithm to ensure the integrity and uniqueness of information.

[0052] Metadata: Data used to describe data, which provides various properties, history, origin, version, and other information about the data.

[0053] In the storage field, deduplication technology has a wide range of applications and its importance is self-evident. By identifying and eliminating redundant parts in data, this technology reduces the space required for data storage and reduces the bandwidth occupied, thereby effectively improving storage efficiency and data transmission efficiency. Data backup is one of the most common application scenarios of deduplication technology. During the backup process, data mainly undergoes a sequential read and write process, because data backup rarely involves random data writing, which enables deduplication technology to exert its maximum efficiency in this scenario. Through data deduplication technology, redundant data can be quickly identified and eliminated, thereby greatly simplifying the complexity of data storage and providing convenience for operations such as duplicate backup. In addition, with the surge in data volume and the continuous expansion of storage requirements, the demand for data transmission bandwidth has also increased. In this case, efficient deduplication technology is particularly important. Therefore, efficient and accurate deduplication technology can not only significantly save storage space, enable the system to back up more data, and increase the capacity of data storage, but also effectively reduce the bandwidth occupied by data transmission and improve data transmission efficiency.

[0054] In actual applications, deduplication technology usually uses a fixed deduplication granularity. For example, most storage device manufacturers will set a fixed deduplication data granularity for users when initially creating storage data. However, due to the constraints of multiple factors such as manufacturing process, software algorithm optimization degree, and actual data deduplication rate, this single fixed granularity setting often cannot achieve the best deduplication efficiency, and this fixed deduplication granularity setting may cause deviation and instability in data output, thereby affecting the accuracy of test results.

[0055] In order to solve the above problems, the present invention provides a data processing method that can efficiently and accurately determine the optimal deduplication granularity. It should be noted that this method is applicable to devices that use deduplication functions in storage arrays such as storage and servers. For details, please refer to Figure 1 , Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention. The method may include:

[0056] S101: Obtain deduplication granularity interval.

[0057] The executor of this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can complete the operation of the data processing method. It should be noted that the deduplication granularity interval obtained in step S101 is the initial deduplication granularity interval. This embodiment does not limit the specific method for determining the initial deduplication granularity interval. For example, the initial deduplication granularity interval can be determined based on the front-end business data; or the initial deduplication granularity interval can also be determined based on the data writing situation.

[0058] Furthermore, in order to determine the accuracy of the initial deduplication granularity interval and improve the efficiency of determining the subsequent optimal deduplication granularity interval, the above-mentioned acquisition of the deduplication granularity interval may specifically include the following steps:

[0059] Step 11: Determine the minimum deduplication granularity and the maximum deduplication granularity according to the current write granularity; the minimum deduplication granularity is smaller than the current write granularity;

[0060] Step 12: Determine the deduplication granularity range according to the minimum deduplication granularity and the maximum deduplication granularity.

[0061] Specifically, when this embodiment writes data into the storage space, the write granularity of the current data can be obtained, and the deduplication granularity interval can be determined according to the current write granularity. For example, when the write granularity of the data is 8K, the minimum deduplication granularity of the initial deduplication granularity interval can be set to 4k, and the maximum deduplication granularity can be set to 128k. It should be noted that in order to improve the accuracy of deduplication, when the deduplication granularity interval contains a deduplication granularity greater than the write granularity, it should also contain a deduplication granularity less than the write granularity. Therefore, the minimum deduplication granularity of the deduplication granularity interval should be less than the write granularity. When the write granularity changes, the deduplication granularity interval determined based on the write granularity will also change accordingly.

[0062] S102: Calculate the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval.

[0063] In this embodiment, the deduplication ratio corresponding to each deduplication granularity is calculated based on the deduplication granularity interval obtained in step S101, and the deduplication granularity interval is adjusted according to the deduplication ratio, so as to determine the optimal deduplication granularity interval. It can be understood that the deduplication ratio in this embodiment is the ratio of the amount of duplicate data deleted in the deduplication data deduplication technology to the total amount of original data. This embodiment can calculate the deduplication ratio corresponding to each deduplication granularity in the deduplication granularity interval; it can also calculate the deduplication ratio corresponding to several deduplication granularities in the deduplication granularity interval.

[0064] Furthermore, in order to reduce computing resource consumption and improve efficiency, the above calculation of the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval and adjusting the deduplication granularity interval according to the deduplication ratio to obtain the optimal deduplication granularity interval may specifically include:

[0065] Step 21: Calculate the deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity, and the middle deduplication granularity in the deduplication granularity interval respectively;

[0066] Step 22: Update the deduplication granularity interval according to the deduplication ratio;

[0067] Repeat the above steps 21 and 22 until the difference between the maximum deduplication granularity and the minimum deduplication granularity of the updated deduplication granularity interval meets the preset threshold, stop updating, and obtain the optimal deduplication granularity interval;

[0068] The above step 22 updates the deduplication granularity interval according to the deduplication ratio, and the specific steps may include:

[0069] The deduplication granularity corresponding to the calculated maximum deduplication ratio is used as the maximum deduplication granularity of the deduplication granularity interval, and the deduplication granularity corresponding to the calculated second largest deduplication ratio is used as the minimum deduplication granularity of the deduplication granularity interval.

[0070] To better understand the process of determining the optimal deduplication granularity interval, you can refer to the following example: the deduplication granularity interval is [a, b], and the deduplication ratios of a, b, and m (m=(a+b) / 2) are calculated and compared; if the deduplication ratio of m> the deduplication ratio of a> the deduplication ratio of b, the deduplication granularity interval is updated to [a, m], and the deduplication ratios of a, m, and m' (m'=(a+m) / 2) are calculated and compared; the above process is repeated and iterated until the two closest deduplication granularities c and d (d>c) are found, and the difference between the two reaches the preset threshold. If the difference between the two is less than 1k, the iteration is stopped, and the optimal deduplication granularity interval [c, d] is obtained. Compared with the method of calculating all deduplication granularities within the deduplication granularity interval, this method of iteratively updating the deduplication granularity interval reduces the amount of calculation, saves computing resources, and improves efficiency.

[0071] Furthermore, in order to improve the calculation efficiency and reduce the calculation resources, after the step 21: respectively calculating the deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity and the middle deduplication granularity in the deduplication granularity interval, the following steps may also be included:

[0072] Determine whether the maximum deduplication ratio among the deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity, and the intermediate deduplication granularity in the deduplication granularity interval reaches a preset deduplication ratio threshold;

[0073] If yes, the deduplication granularity corresponding to the maximum deduplication ratio is used as the optimal deduplication granularity;

[0074] If not, continue to execute the step of updating the deduplication granularity interval according to the deduplication ratio.

[0075] For details, please refer to Figure 2 , Figure 2An example diagram of an iterative optimization process provided by an embodiment of the present invention. In the process of calculating the deduplication ratio, this embodiment adds a preset deduplication ratio threshold, and compares the maximum deduplication ratio in the interval with the preset deduplication ratio threshold. If it is greater than the preset deduplication ratio threshold, the loop ends early, and the deduplication granularity corresponding to the maximum deduplication ratio is used as the optimal deduplication granularity; if it is less than the preset deduplication ratio threshold, the interval is updated, the deduplication ratio of the new interval is recalculated, and the maximum deduplication ratio in the new interval is compared with the preset deduplication ratio threshold, and so on. If there is no deduplication granularity greater than the preset deduplication ratio threshold during the iterative optimization process, the optimal deduplication granularity is selected from the optimal deduplication granularity interval. In this way, the iterative optimization process can be reduced, the optimal deduplication granularity can be found, the efficiency of iterative optimization can be improved, computing resources can be saved, and the system can be kept running efficiently and stably.

[0076] S103: Determine an optimal deduplication granularity from the optimal deduplication granularity range, and delete duplicate data in the storage space according to the optimal deduplication granularity.

[0077] This embodiment does not limit the specific method of determining the optimal deduplication granularity based on the optimal deduplication granularity interval. For example, the deduplication ratios corresponding to the various deduplication granularities in the optimal deduplication granularity interval may be calculated, and the deduplication granularity corresponding to the maximum deduplication ratio may be used as the optimal deduplication granularity; or a certain deduplication granularity in the optimal deduplication granularity interval may be randomly selected as the optimal deduplication granularity.

[0078] Further, in order to improve the efficiency and accuracy of determining the optimal deduplication granularity, the above-mentioned determination of the optimal deduplication granularity from the optimal deduplication granularity interval may specifically include:

[0079] The middle value of the optimal deduplication granularity range is taken as the optimal deduplication granularity.

[0080] It is understandable that the difference in deduplication granularity within the optimal deduplication granularity interval is small, and the interval range is small, so the difference in deduplication ratios corresponding to each deduplication granularity is small. Therefore, the middle value in the interval can be directly used as the optimal deduplication granularity, which can reduce the calculation process and ensure accuracy.

[0081] It should be noted that the data deduplication technology can be performed immediately when the data is written to the storage space, or it can be selected to perform data deduplication through a scheduled task after the data is written to the storage space. Furthermore, the timing method is more conducive to the accuracy of determining the granularity of deduplication. Therefore, this embodiment can perform data deduplication in a timing manner, thereby more accurately identifying and eliminating duplicate data. Especially for long data, the effect is more obvious. Furthermore, it is also possible to first split the data after writing, and then perform deduplication processing of subdivided granularity on the split data blocks to achieve a more optimized effect.

[0082] Furthermore, the above method may also include:

[0083] Generate a data analysis report at each deduplication granularity, the data analysis report at least including statistical data on write granularity, deduplication ratio, saved storage capacity and deduplication data volume within a preset time.

[0084] Specifically, at each deduplication granularity, detailed data analysis reports can be generated, including statistics on data write granularity, statistics on deduplication ratio, storage capacity saved after deduplication, and amount of deduplicated data, for comprehensive analysis and comparison.

[0085] Further, after step S103, the above method may further include the following steps:

[0086] Step 31: Obtain the data fingerprint of the data written into the storage space, and cache the data fingerprint in the storage space; the data fingerprint is set according to the data segment structure;

[0087] Step 32: Write the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system.

[0088] For details, please refer to Figure 3 , Figure 3 The flowchart of a data processing method provided by an embodiment of the present invention is shown in FIG. When writing data into a storage space, the corresponding data fingerprint is also cached to the storage space for use when the subsequent data is stored on the disk. Through the above-mentioned effective data deduplication technology, the space of the hard disk can be saved when the data is stored on the disk. It should be noted that step 31 is a data fingerprint cache pre-fetching process. The data fingerprint in this embodiment is set according to the data segment structure. Because the data segment structure of the data written into the storage space is inconsistent, it can be used as a data fingerprint. The metadata management system manages the data in a standardized, standardized and unified manner to improve the reliability and maintainability of the data. The metadata management system can also perform standardized management on the data fingerprint to ensure the consistency and accuracy of the data, and also facilitate data tracking and auditing to determine the integrity and authenticity of the data. The metadata management system can also provide security measures such as data encryption and access control to protect the security and privacy of the data. In this embodiment, digital fingerprint technology and metadata management system can be used to ensure the integrity, authenticity and traceability of the data, which can further improve the efficiency and reliability of data processing. The metadata management system takes advantage of the locality of data and introduces data fingerprint cache pre-fetching, which reduces data interference and achieves orderly disk placement, which is beneficial for subsequent data reading and can increase the disk placement speed, greatly improving the deduplication data throughput.

[0089] Furthermore, the above step 32 writes the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system, which may specifically include the following steps:

[0090] When the hard disk is empty, all remaining data will be written to the hard disk according to the data fingerprint;

[0091] When the hard disk is not empty, the remaining data is compared with the data in the hard disk for similarity based on the data fingerprint to determine the same data and different data in the remaining data;

[0092] Write different data to the hard disk.

[0093] Specifically, when the hard disk is empty, it means that the hard disk has sufficient capacity and no data. At this time, all the remaining data in the storage space after deduplication needs to be written to the disk. When there is data on the hard disk, the data can be compared for similarity based on the data fingerprint pre-fetched from the cache. For details, please refer to Figure 4 , Figure 4 This is an example diagram of a data write process provided by an embodiment of the present invention. The remaining data in the storage space is compared with the data in the hard disk for similarity. If the same data is found, the original marked position is used to quickly find the complete data and the number of writes is recorded; if not found, the data is written to the hard disk and the number of writes is recorded.

[0094] Furthermore, the above step 32 writes the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system, which may specifically include the following steps:

[0095] Step 321: Obtain the number of write times, and divide the remaining data into hot data and cold data according to the number of write times;

[0096] Step 322: write the hot data to the center of the disk according to the data fingerprint, and write the cold data to the edge of the disk according to the data fingerprint.

[0097] Specifically, hot data is data with high access frequency and critical to business and applications; cold data is data with low access frequency and not very important to business and applications. This embodiment can divide data into hot and cold data according to the data writing situation. For example, in a 24-hour cycle, hot and cold data are distinguished according to the number of recorded data writes. Hot data with frequent read and write is stored nearby on the hard disk, and cold data with less read and write is stored at the edge. This can significantly improve the write speed of data deduplication. When data needs to be read from the hard disk, data with similar fingerprints can be quickly found from a location with higher heat. This mechanism not only improves the efficiency of deduplication, but also speeds up the data refresh rate, making full use of the advantages of hard disk storage location and similarity comparison, as well as the effect of recording the number of writes on improving data management efficiency.

[0098] The data processing method provided by the embodiment of the present invention is applied to obtain the deduplication granularity interval; calculate the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain the optimal deduplication granularity interval; determine the optimal deduplication granularity from the optimal deduplication granularity interval, and delete duplicate data in the storage space according to the optimal deduplication granularity. The present invention adaptively adjusts the deduplication granularity interval through the deduplication ratio to obtain the optimal deduplication granularity interval, and further determines the optimal deduplication granularity from the optimal deduplication granularity interval, and uses the optimal deduplication granularity to perform data deduplication. This flexible and accurate deduplication granularity determination method can not only ensure the normal operation of data deduplication, but also significantly improve the deduplication ratio, save storage space, further optimize deduplication performance, and quickly store data on disk. In addition, this embodiment uses the number of write times to divide data into hot and cold data, and facilitates the rapid flushing of data based on the statistics of hot and cold data; and, caches the local features (data fingerprints) of pre-fetched data, reduces data interference, can quickly and orderly implement data flushing, and improves the throughput of deduplicated data; and, generates periodic reports to conduct in-depth analysis of business and application data, and facilitates the analysis and organization of optimized deduplication granularity; this method can achieve larger-scale and more efficient data storage, further reduce costs, and increase benefits.

[0099] The following is an introduction to a data processing method and apparatus provided in an embodiment of the present invention. The data processing method and apparatus described below and the data processing method described above can be referenced to each other.

[0100] Please refer to Figure 5 , Figure 5 A structural diagram of a data processing method and device provided by an embodiment of the present invention may include:

[0101] A deduplication ratio interval acquisition module 100 is used to obtain a deduplication granularity interval;

[0102] The optimal deduplication granularity interval determination module 200 is used to calculate the deduplication ratio corresponding to the deduplication granularity within the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain the optimal deduplication granularity interval;

[0103] The deduplication module 300 is used to determine an optimal deduplication granularity from the optimal deduplication granularity interval, and delete duplicate data in the storage space according to the optimal deduplication granularity.

[0104] Based on the above embodiment, the deduplication ratio interval obtaining module 100 may include:

[0105] a granularity determination unit, configured to determine a minimum deduplication granularity and a maximum deduplication granularity according to a current write granularity; the minimum deduplication granularity is smaller than the current write granularity;

[0106] An interval determining unit, configured to determine the deduplication granularity interval according to the minimum deduplication granularity and the maximum deduplication granularity

[0107] Based on any of the above embodiments, the optimal deduplication granularity interval determination module 200 may include:

[0108] a deduplication ratio calculation unit, configured to respectively calculate the deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity, and the middle deduplication granularity in the deduplication granularity interval;

[0109] an interval updating unit, configured to update the deduplication granularity interval according to the deduplication ratio;

[0110] an iterative unit, configured to repeatedly execute the steps of respectively calculating the deduplication ratios at the minimum deduplication granularity, the maximum deduplication granularity, and the intermediate deduplication granularity in the deduplication granularity interval and updating the deduplication granularity interval according to the deduplication ratios, until the difference between the maximum deduplication granularity and the minimum deduplication granularity of the updated deduplication granularity interval meets a preset threshold, stop updating, and obtain the optimal deduplication granularity interval;

[0111] Wherein, the interval updating unit may include:

[0112] The interval updating subunit is used to use the deduplication granularity corresponding to the calculated maximum deduplication ratio as the maximum deduplication granularity of the deduplication granularity interval, and use the deduplication granularity corresponding to the calculated second largest deduplication ratio as the minimum deduplication granularity of the deduplication granularity interval.

[0113] Based on the above embodiment, the deduplication module 300 may include:

[0114] The optimal deduplication granularity determining unit is used to use the middle value of the optimal deduplication granularity interval as the optimal deduplication granularity.

[0115] Based on the above embodiment, the data processing device may further include:

[0116] A data fingerprint processing module, used to obtain a data fingerprint of the data written into the storage space, and cache the data fingerprint into the storage space; the data fingerprint is set according to the data segment structure;

[0117] The data writing module is used to write the remaining data after deletion in the storage space into the hard disk according to the data fingerprint and metadata management system.

[0118] Based on the above embodiment, the data storage module may include:

[0119] A first writing unit, configured to write all the remaining data into the hard disk according to the data fingerprint when the hard disk is empty;

[0120] The second writing unit is used for comparing the remaining data with the data in the hard disk for similarity according to the data fingerprint when the hard disk is not empty, determining the same data and different data in the remaining data; and writing the different data into the hard disk.

[0121] Based on the above embodiment, the data storage module may include:

[0122] A data partitioning unit, used for obtaining the number of write times, and partitioning the remaining data into hot data and cold data according to the number of write times;

[0123] The third writing unit is used to write the hot data to the center position of the disk according to the data fingerprint, and write the cold data to the edge position of the disk according to the data fingerprint.

[0124] It should be noted that the order of the modules and units in the above data processing device can be changed without affecting the logic.

[0125] The data processing device provided by the embodiment of the present invention is applied, and is used to obtain the deduplication granularity interval through the deduplication ratio interval acquisition module 100; the optimal deduplication granularity interval determination module 200 is used to calculate the deduplication ratio corresponding to the deduplication granularity in the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain the optimal deduplication granularity interval; the deduplication module 300 is used to determine the optimal deduplication granularity from the optimal deduplication granularity interval, and delete duplicate data in the storage space according to the optimal deduplication granularity. The present invention adaptively adjusts the deduplication granularity interval through the deduplication ratio to obtain the optimal deduplication granularity interval, and further determines the optimal deduplication granularity from the optimal deduplication granularity interval, and uses the optimal deduplication granularity to perform data deduplication. This flexible and accurate deduplication granularity determination method can not only ensure the normal operation of data deduplication, but also significantly improve the deduplication ratio, save storage space, further optimize the deduplication performance, and quickly store data on disk. In addition, this embodiment uses the number of write times to divide data into hot and cold data, and facilitates the rapid flushing of data based on the statistics of hot and cold data; and, caches the local features (data fingerprints) of pre-fetched data, reduces data interference, can quickly and orderly implement data flushing, and improves the throughput of deduplicated data; and, generates periodic reports to conduct in-depth analysis of business and application data, and facilitates the analysis and organization of optimized deduplication granularity; this method can achieve larger-scale and more efficient data storage, further reduce costs, and increase benefits.

[0126] Figure 6 A structural diagram of a data processing device provided by an embodiment of the present invention, such as Figure 6 As shown, the data processing device includes:

[0127] A memory 60, for storing computer programs;

[0128] The processor 61 is used to implement the steps of the data processing method in the above embodiment when executing a computer program.

[0129] The data processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0130] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0131] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein the computer program can implement the relevant steps of the data processing method disclosed in any of the aforementioned embodiments after being loaded and executed by the processor 61. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include, but is not limited to, data of the data processing method, etc.

[0132] In some embodiments, the data processing device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0133] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation of the data processing device, and may include more or fewer components than shown in the figure.

[0134] It is understandable that if the data processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0135] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0136] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be cross-referenced with other embodiments described in this document.

[0137] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0138] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0139] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0140] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0141] The above is a detailed introduction to a data processing method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: include: Get the deduplication granularity range; Calculating a deduplication ratio corresponding to a deduplication granularity within the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval; An optimal deduplication granularity is determined from the optimal deduplication granularity interval, and duplicate data in the storage space is deleted according to the optimal deduplication granularity.

2. The data processing method according to claim 1, characterized in that: Get the deduplication granularity range, including: Determine a minimum deduplication granularity and a maximum deduplication granularity according to the current write granularity; the minimum deduplication granularity is smaller than the current write granularity; The deduplication granularity interval is determined according to the minimum deduplication granularity and the maximum deduplication granularity.

3. The data processing method according to claim 1, characterized in that: Also includes: Obtaining a data fingerprint of the data written into the storage space, and caching the data fingerprint in the storage space; The data fingerprint is set according to the data segment structure; The remaining data after deletion from the storage space is written into the hard disk according to the data fingerprint and metadata management system.

4. The data processing method according to claim 3, characterized in that: Writing the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system includes: When the hard disk is empty, all the remaining data are written into the hard disk according to the data fingerprint; When the hard disk is not empty, the remaining data is compared with the data in the hard disk for similarity according to the data fingerprint to determine the same data and different data in the remaining data; The different data are written into the hard disk.

5. The data processing method according to claim 3, characterized in that: Writing the remaining data after deletion in the storage space to the hard disk according to the data fingerprint and metadata management system includes: Obtaining the number of write times, and dividing the remaining data into hot data and cold data according to the number of write times; The hot data is written to the center of the disk according to the data fingerprint, and the cold data is written to the edge of the disk according to the data fingerprint.

6. The data processing method according to any one of claims 1 to 5, characterized in that: Calculating a deduplication ratio corresponding to a deduplication granularity within the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval, including: Calculating deduplication ratios corresponding to the minimum deduplication granularity, the maximum deduplication granularity, and the middle deduplication granularity in the deduplication granularity interval respectively; updating the deduplication granularity interval according to the deduplication ratio; Repeat the steps of respectively calculating the deduplication ratios at the minimum deduplication granularity, the maximum deduplication granularity, and the intermediate deduplication granularity in the deduplication granularity interval and updating the deduplication granularity interval according to the deduplication ratios until the difference between the maximum deduplication granularity and the minimum deduplication granularity of the updated deduplication granularity interval meets a preset threshold, stop updating, and obtain the optimal deduplication granularity interval; Wherein, updating the deduplication granularity interval according to the deduplication ratio includes: The deduplication granularity corresponding to the calculated maximum deduplication ratio is used as the maximum deduplication granularity of the deduplication granularity interval, and the deduplication granularity corresponding to the calculated second largest deduplication ratio is used as the minimum deduplication granularity of the deduplication granularity interval.

7. The data processing method according to claim 6, characterized in that: Determining an optimal deduplication granularity from the optimal deduplication granularity range includes: The middle value of the optimal deduplication granularity interval is used as the optimal deduplication granularity.

8. A data processing device, characterized in that: include: A deduplication ratio interval acquisition module is used to obtain a deduplication granularity interval; an optimal deduplication granularity interval determination module, configured to calculate a deduplication ratio corresponding to a deduplication granularity within the deduplication granularity interval, and adjust the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval; The deduplication module is used to determine the optimal deduplication granularity from the optimal deduplication granularity interval, and delete duplicate data in the storage space according to the optimal deduplication granularity.

9. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images