Data compression method, apparatus and device

Abstract

The application provides a data compression method, device and equipment. The method comprises the following steps: obtaining data to be stored; determining a target compression dictionary according to a classification result of the data; and compressing the data according to the target compression dictionary to obtain compressed data. The application effectively improves the service life and access performance of a storage device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, in particular to a data compression method, device and equipment. BACKGROUND

[0002] The storage medium in a solid state disk (SSD) will be worn out during use, and in a non-volatile memory, multiple pieces of bit information are stored in a cell, read and written in units of pages, and erased in units of blocks.

[0003] In the related art, a storage device can only receive storage-related information in an input / output command, and the amount of data written and read is large, and when the number of erasing and writing times of a block reaches a limit, the error rate is greatly increased, which seriously affects the service life and access performance of the solid state disk. SUMMARY

[0004] The present application provides a data compression method, device and equipment, which classifies the data to be stored, so that the same type of data can be compressed using the same dictionary file, effectively reducing the number of dictionary file entries, and realizing low-overhead data compression. The compression dictionary determined based on the classification result of the data is used to compress and store the data, so that the storage space occupied by the same data is less, more data can be stored on the limited storage medium, the amount of data written and read is reduced, and the service life and access performance of the storage device are improved.

[0005] The present application provides a data compression method, comprising the following steps.

[0006] Obtaining data to be stored;

[0007] According to the classification result of the data, a target compression dictionary is determined;

[0008] According to the target compression dictionary, the data is compressed to obtain compressed data.

[0009] According to the data compression method provided by the present application, the method further comprises:

[0010] According to the data classification model, the classification result of the data is determined; the data classification model is used to identify the type of the data to be stored.

[0011] According to the data compression method provided by the present application, the method further comprises:

[0012] The compression category information is stored in a first flash translation layer (FTL) mapping table.

[0013] The storage location of the compressed data and the size of the compressed data are stored in the second FTL mapping table.

[0014] According to the data compression method provided by the application, the method further comprises:

[0015] In the case that the size of the data to be stored is greater than the first value and less than the second value, if the sum of the compressed data and the uncompressed data is less than the first value during the compression of the data to be stored, the compression of the uncompressed data is stopped and the compressed data and the uncompressed data are stored; the size of the first value is related to the size of the physical page of the solid state disk.

[0016] According to the data compression method provided by the application, the method further comprises:

[0017] In the case that the size of the data to be stored is greater than the second value, the threshold value of stopping compression corresponding to the data to be stored is determined according to the size of the sequential write data amount and the size of the random write data amount in the data to be stored.

[0018] According to the data compression method provided by the application, the threshold value of stopping compression corresponding to the data to be stored is determined according to the size of the sequential write data amount and the size of the random write data amount in the data to be stored, comprising:

[0019] The threshold value of stopping compression corresponding to the data to be stored is determined based on the following manner:

[0020]

[0021] Wherein, The threshold value of stopping compression corresponding to the data to be stored is represented; The size of the sequential write data amount in the data to be stored is represented; r represents the size of the random write data amount in the data to be stored; p represents the size of the physical page in the solid state disk; n represents a preset multiple value of the size of the physical page; Related to the size of the compression dictionary.

[0022] The application further provides a data compression device, comprising the following modules:

[0023] The acquisition module is used for acquiring data to be stored;

[0024] The determination module is used for determining a target compression dictionary corresponding to the data according to the classification result of the data;

[0025] The compression module is used for compressing the data according to the target compression dictionary to obtain compressed data.

[0026] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data compression method according to any one of the above when executing the program.

[0027] The application further provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the data compression method according to any one of the above.

[0028] The application further provides a computer program product, comprising a computer program, wherein the computer program is executable on a processor to implement the data compression method according to any one of the above.

[0029] The data compression method, device and electronic device provided by the application can effectively reduce the dictionary file entries by classifying the data to be stored, so that the same type of data can be compressed by using the same dictionary file, and realize low-overhead data compression. Moreover, the compression dictionary determined based on the classification result of the data is used to compress and store the data, so that the storage space occupied by the same data is less, more data can be stored on the limited storage medium, the data amount of writing and reading is reduced, and the service life and access performance of the storage device are improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0031] Figure 1 is one of the flowcharts of the data compression method provided by the application.

[0032] Figure 2 is a schematic diagram of the mapping table provided by the application.

[0033] Figure 3 is a schematic diagram of the compression module provided by the application.

[0034] Figure 4 is the second flowchart of the data compression method provided by the application.

[0035] Figure 5 is a schematic diagram of the data compression device provided by the application.

[0036] Figure 6 is a schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0037] So that the objects, technical solutions and advantages of the present application are more apparent, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] The data compression method, device and equipment of the present application are described below. Figures 1-6 The data compression method, device and equipment of the present application are described below.

[0039] In order to more clearly understand the technical solutions of the embodiments of the present application, first, some technical contents related to the embodiments of the present application are introduced.

[0040] The Flash storage medium in the solid state disk (SSD) will be worn out when used. In the NAND memory, multiple bit information is stored in a cell, read and written in page units, and erased in block units. For a storage cell of TLC, it will start to wear out after 3000 write-erase operations. When the erase-write times of a block reach the limit, the error rate will be greatly increased, which seriously affects the service life of the solid state disk. Therefore, in the system design aspect, the write and erase times should be reduced as much as possible, and the service life of the solid state disk should be maximally prolonged without increasing the cost of the hard disk.

[0041] The data compression technology is a means to prolong the service life of the solid state disk. In the same condition, the less data written, the less erase times. Therefore, the data compression can be deployed inside the SSD, and the received user data is first compressed and then stored in the NAND, which can effectively reduce the data size and erase-write times. In addition, after data compression, the actual write and read data amount is reduced, and the access performance of the SSD can also be improved. The data compression algorithm reduces the data amount by eliminating the redundant information in the data, thereby achieving the purpose of data compression. The dictionary compression technology is one of the common data compression algorithms. Before compression, a dictionary is first constructed, which contains the common patterns of the data to be compressed. When compressing the data, the data to be compressed is matched with the dictionary, and the matched pattern is replaced with the corresponding index value. Finally, the compressed data and the dictionary are saved for decompression operation.

[0042] Many file systems allow users to turn on file compression to save space needed on storage devices. But these file system level compression mechanisms consume valuable CPU resources. Meanwhile, storage devices can only receive storage related information in I / O commands and cannot use context related advanced information (such as file type) to avoid compressing files that are already in compressed format, and cannot use optimized algorithms for different file types. Meanwhile, existing methods build dictionaries for different data blocks separately, inevitably leading to dictionary duplication.

[0043] Figure 1 is one of the flowcharts of the data compression method provided by the present application, as shown in Figure 1 The method comprises the following steps:

[0044] Step 101, obtaining data to be stored.

[0045] Specifically, in the prior art, storage devices can only receive storage related information in input / output commands, and the amount of data written and read is large, resulting in poor access performance of solid state disks. In order to solve the above problems, in the embodiments of the present application, data to be stored is first obtained. Optionally, in the embodiments of the present application, the storage device can be connected with a host, and the host sends a storage instruction to the storage device, instructing the storage device to store data.

[0046] Step 102, determining a corresponding target compression dictionary according to the classification result of the data.

[0047] Specifically, after obtaining the data to be stored, the storage device can classify the data to be stored, and determine a corresponding target compression dictionary according to the classification result of the data to perform compression. That is, in the embodiments of the present application, the same dictionary file is used for data belonging to the same category, so that the number of dictionary file entries can be effectively reduced, and low-overhead data compression can be achieved. Optionally, the construction of the compression dictionary can be based on existing methods, which will not be described in detail in the embodiments of the present application.

[0048] Step 103, compressing the data according to the target compression dictionary to obtain compressed data.

[0049] Specifically, after determining the corresponding target compression dictionary according to the classification result of the data, the data can be compressed and stored based on the determined compression dictionary, so that the storage space occupied by the same data is less, more data can be stored on the limited storage medium, the amount of data written and read is reduced, and the service life and access performance of the storage device are improved.

[0050] The method of the above embodiment can effectively reduce the dictionary file entries by classifying the data to be stored, so as to compress the data of the same category by using the same dictionary file, and realize low-overhead data compression. Moreover, the data is compressed and stored based on the compression dictionary determined according to the classification result of the data, so that the storage space occupied by the same data is less, more data can be stored on the limited storage medium, the data amount of writing and reading is reduced, and the service life and access performance of the storage device are improved.

[0051] In an embodiment, the data compression method further comprises:

[0052] According to the data of the preset size in the data to be stored and the preset data classification model, a classification result of the data is determined. The data classification model is used to identify the type of the data to be stored.

[0053] Specifically, when classifying the data to be stored, the present application only needs to scan a small amount of write data to determine whether to compress and the compression category, so as to effectively improve the response speed.

[0054] Optionally, in the present application, the classification of the data to be stored is based on the preset data classification model, which can effectively utilize the technical advantages of machine learning to realize the classification and compression of the stored data, reduce the repeated entries of the compression dictionary and the space occupied by the data on the storage medium, and improve the storage capacity.

[0055] For example, the classification model in the present embodiment is as follows:

[0056] (1) The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method is used to cluster the sample data with high similarity, and determine the optimal number of categories of the sample data.

[0057] (2) The sample data is marked with the category number generated by clustering, and the marked sample data is input into the KNN classification model for classification training. The sample features include random and continuous two data command request modes, the number of LBAs, the type of LBAs (LBAf), the proportion of mixed read and write in data I / O, etc. Based on a certain distance measurement, the K closest training samples in the training set to the test set sample are found, and then the K training samples are used to determine the category of the test sample by using the "voting method".

[0058] Optionally, the compression program uses a dictionary-based algorithm to convert the input data into 16-bit unsigned integers, and calculates the occurrence number of each integer to generate the optimal dictionary (the cache version of the dictionary is maintained by the DRAM space of the SSD).

[0059] Optionally, among the numerous clustering algorithms, a density-based clustering method (DBSCAN) is selected for use, which can automatically determine the number of cluster centers without human intervention. In addition, some files, such as bmp files, do not support file compression, and jpg (jpeg) files can only be compressed with loss, so the clustering algorithm will classify data types that are not suitable for compression into the same category, and the resulting model can also be used to predict whether the input data has compressible value. The classification model uses the k-nearest neighbor algorithm (KNN) which has high accuracy, is not sensitive to outliers, has no data input assumption, and supports multi-classification. KNN is a supervised learning algorithm that determines the class of a new sample according to the classification decision of the k training samples closest to the new sample. The selection of distance measure, k value and classification decision rule are the three basic elements of k-nearest neighbor method. Since modern file systems tend to assign consecutive logical block addresses to the same file, part of the data (512 bytes) is read from the write request for classification prediction.

[0060] Compared with the method of generating training samples by manually labeling a large amount of data, the method of the embodiments of the present application clusters sample data using a clustering algorithm, which is time-consuming and can avoid errors in generating training samples. The sample data used for training includes text files, program binary files, pictures, videos, program source code, etc. Using a clustering algorithm avoids the error-prone and time-consuming manual labeling sample method for a large amount of data. In addition, since the compression module uses idle cores for prediction and compression processing, the classification model should make a prediction within the delay time of writing to the flash page, so that the firmware program uses pipeline parallelism to hide the prediction delay in the worst case. Generally, the minimum page write delay is about 200us, so the training goal of the classification model is to achieve classification within 200us.

[0061] The method of the above embodiments classifies the data to be stored based on a preset data classification model, which can effectively utilize the technical advantages of machine learning to realize the classification and compression of stored data, reduce the number of repeated entries in the compression dictionary and the space occupied by the data on the storage medium, and improve the storage capacity. Moreover, when classifying the data to be stored, only a small amount of write data needs to be scanned to determine whether to compress and the compression category, which can effectively improve the response speed.

[0062] In an embodiment, the data compression method further comprises:

[0063] Storing the compression category information in a first flash translation layer (FTL) mapping table;

[0064] Storing the storage location of the compressed data and the size of the compressed data in a second FTL mapping table.

[0065] Specifically, in a conventional flash memory storage system, each physical flash page corresponds to the same number of logical data blocks. However, after data compression, the flash pages of an SSD can correspond to different numbers of logical blocks, and therefore a new mapping table is needed to record the compressed data information, and a field is added to the original mapping table to distinguish compressed data from uncompressed data. Alternatively, embodiments of the present application store compressed category information in a first flash translation layer (FTL) mapping table and store the storage location and size of compressed data in a second FTL mapping table, thereby accurately recording the compressed data information, so that the stored compressed data can be accurately and quickly located when reading the data in the storage device.

[0066] For example, Figure 2 For the FTL mapping table, a first FTL mapping table and a second FTL mapping table are included.

[0067] Alternatively, the first FTL mapping table adds a compressed category field to the original mapping table, saves the mapping between the logical page number (LPN) and the physical page number (PPN), and records the compressed category. The category of 0 represents that the data is not compressed. In the first FTL mapping table, LPN(0) and LPN(1) are mapped to PPN(26) and PPN(44), and are associated with uncompressed data; LPN(16), LPN(17), LPN(18), and LPN(19) are mapped to PPN(89), and LPN(42), LPN(43), and LPN(44) are mapped to PPN(90), and are associated with compressed data.

[0068] Alternatively, the second FTL mapping table records the information of the compressed data stored on the physical page, including the offset (OFF) and the length (LEN). The offset field saves the starting position in the corresponding physical address, and the length field saves the length of the compressed data. When the compressed data is stored in a separate physical page, it can cause a part of the page to be wasted. Therefore, when writing compressed data continuously, the next sub-page after the previous compressed data can be written to, thereby avoiding the waste of space. The combination of the offset field and the length field can achieve this storage method.

[0069] For example, if the physical page size is 8KB, it can be divided into 16 sub-pages (sub-page size 512B). The compressed data can be written from the 0th sub-page of PPN(89) first, and since the data length is 10000B, it can be written to the 4th sub-page of PPN(90). Although 240 bytes of the 4th sub-page of PPN(90) cannot be avoided, the wasted space is very small compared to the single-page storage of the page compression. The next compressed data can be written from the 5th sub-page of PPN(90), and therefore the offset is 5, that is, the offset value represents the starting physical sub-page of the stored compressed data.

[0070] Optionally, when the SSD receives a read command, the first mapping table is first queried to obtain the requested physical page code from the flash array, and it is determined according to the compression category field whether the data to be read is compressed. For compressed data, the second mapping table is queried to obtain the compressed data according to the physical page code, offset and length. Then, the original data content is reconstructed using the dictionary file corresponding to the compression category and is decompressed into the SSD DRAM buffer. After the decompression is completed, the I / O interface starts the DMA data transmission and completes the request.

[0071] The method of the above embodiment stores the compression category information in the first flash translation layer (FTL) mapping table and stores the storage location of the compressed data and the size of the compressed data in the second FTL mapping table, so that the accurate recording of the compressed data information is quickly and accurately realized, and thus the stored compressed data can be accurately and quickly located when the data in the storage device is read.

[0072] In an embodiment, the data compression method further comprises:

[0073] In a case where the size of the data to be stored is greater than a first value and less than a second value, if it is determined that the sum of the compressed data and the uncompressed data is less than the first value in the compression process of the data to be stored, the compression of the uncompressed data is stopped and the compressed data and the uncompressed data are stored; the size of the first value is related to the size of the physical page of the solid state disk.

[0074] Specifically, in a case where the size of the data to be stored is greater than a first value and less than a second value, if it is determined that the sum of the compressed data and the uncompressed data is less than the first value in the compression process of the data to be stored, the compression of the uncompressed data is stopped and the compressed data and the uncompressed data are stored, so that the response speed to the compression instruction can be effectively improved on the basis of reducing the storage space. Optionally, the size of the first value can be the size of the physical page of the storage device, such as 4K.

[0075] For example, taking the physical page of the storage device as 4K as an example, when the data to be stored is greater than 4K and less than 8K, the amount of compressed data is monitored in the process of pattern matching with the dictionary file, and when it is found that the total data amount is less than 4K (including the compressed data and the uncompressed data), the compression algorithm is stopped, the length of the compressed data is recorded and single-page storage is performed. When the data is read, the compressed part is decompressed and combined with the uncompressed part to generate the original data. Thus, the response speed to the compression instruction and the storage instruction can be effectively improved on the basis of reducing the storage space.

[0076] The method of the above embodiment can stop compression of the uncompressed data and store the compressed data and the uncompressed data when the sum of the compressed data and the uncompressed data is less than the first value during compression of the data to be stored, when the size of the data to be stored is greater than the first value and less than the second value, thereby improving the response speed of the compression instruction and the storage instruction effectively while reducing the storage space.

[0077] In an embodiment, the data compression method further comprises:

[0078] In the case where the size of the data to be stored is greater than the second value, the threshold value of stopping compression corresponding to the data to be stored is determined according to the size of the sequential write data amount and the size of the random write data amount in the data to be stored.

[0079] Specifically, in the case where the size of the data to be stored is greater than the second value, the threshold value of stopping compression corresponding to the data to be stored is determined according to the size of the sequential write data amount and the size of the random write data amount in the data to be stored in the embodiment, so that the compression can be stopped when the size of the compressed data reaches the threshold value of stopping compression, thereby improving the response speed of the compression instruction and the storage instruction effectively while reducing the storage space.

[0080] For example, when the data to be stored is greater than n 4K (n≥2), the decision mechanism is processed, and the threshold value of stopping compression thr is set, the compression is stopped when the data amount is less than the threshold value, and the threshold value is related to the write mode and the dictionary file size {a1, a2, …, aM}:

[0081]

[0082] wherein, the threshold value of stopping compression corresponding to the data to be stored; the size of the sequential write data amount in the data to be stored; r represents the size of the random write data amount in the data to be stored; and p represents the size of the physical page in the solid state disk. That is, in the case where the sequential write data amount is greater than the random write data amount, the compression threshold value is relatively small, so that a larger compression ratio can be obtained; in the case where the random write occupies a high proportion, the compression threshold value is relatively large, so that a faster response can be obtained.

[0083] The method of the above embodiment determines the threshold value of stopping compression corresponding to the data to be stored according to the size of the sequential write data amount and the size of the random write data amount in the data to be stored, so that the compression can be stopped when the data size in the compression process is the threshold value of stopping compression, the response speed to the compression instruction can be effectively improved on the basis of reducing the storage space, and the compression ratio can be larger when the sequential write data amount is larger than the random write data amount, and the compression threshold value is relatively small, so that a larger compression ratio can be obtained, and the response speed is relatively fast when the random write proportion is high.

[0084] Exemplarily, Figure 3 The SSD system architecture comprises a data compression module (MLDC).

[0085] In the SSD system architecture of the present application, in addition to the flash array for data storage, the SSD also comprises a system on chip (SoC) including an I / O interface, a microprocessor kernel, a hardware accelerator, a flash interface, and a DRAM interface as a controller. In order to communicate with the host, the I / O interface is connected to a PCIe, SATA slot, and uses a SATA or NVMe standard to interact with the host. The flash array of the SSD is usually multi-channel, allowing parallel access, and providing sufficient internal access bandwidth between the data array and the SoC-based controller. The embedded kernel inside the Soc will execute a firmware program to parse the commands from the I / O interface and perform the FTL function. The FTL layer completes the mapping of the host logical address (LBA) to the flash (Flash) physical address space (PPN). Most SSDs have on-board DRAM as a buffer for writing data or data cache, and most of the mapping tables can be placed on the DRAM to improve access performance. The flash controller is responsible for managing the reading and writing of data from the cache to the flash.

[0086] The machine learning-based data compression module (MLDC) interacts with the host through the I / O interface, classifies the input data, and realizes lossless compression of the data in combination with the dictionary compression algorithm. The core of the dictionary compression algorithm is to create a dictionary, so that the repeated patterns in the original data are converted into a shorter form. Optionally, the MLDC module specifically comprises a classifier, a dictionary file unit, a data compressor, and a data decompressor. The classifier comprises a well-trained classification model, which can classify input data with the same characteristics into the same compression category, and each category corresponds to a specific dictionary file generated by training. The compressor compresses data using the specified dictionary file. When reading data, the decompressor decompresses data according to the compression category using the corresponding dictionary file.

[0087] Exemplarily, Figure 4 The compression process for input data of a compressed solid state disk is specifically:

[0088] 1. When the host interface receives a write command, it forwards the command to the MLDC module.

[0089] 2. The classifier uses the DMA hardware unit to look at a small portion of the write data in the data buffer, and the prediction model determines the compression class and whether the data is compressible based on this portion of data. That is, the machine learning model combined by the clustering algorithm and the classification algorithm in the embodiment of the present application can classify the input data, and only a small amount of write data needs to be scanned to determine the compression class or data that does not need to be compressed, thereby reducing the computational overhead of the solid state drive.

[0090] 3. The data compressor starts to perform the compression operation:

[0091] If the data belongs to a class that is not compressible or the compression benefit is limited (such as a file that is already in a compressed state), the compression operation is not performed to save computational resources;

[0092] If the data belongs to a compressible class, the compressor will use the shared dictionary file of the class specified by the classifier to match the data to be compressed with the dictionary, replace the corresponding index value with the matched pattern, and compress the entire buffer data content. The classifier uses the same dictionary file for data of the same class to avoid duplication of dictionary entries. Optionally, for compressed data, the read request may require the data decompressor to extract multiple flash pages, such as reading LPN(43), which requires the data decompressor to decompress the entire PPN(90), which can cause read command delay. To alleviate the read delay to some extent, the read request can be responded to immediately when decompression to the offset. In addition, the firmware program designed for the MLDC module retains the decompressed data in the SSD DRAM buffer until the buffer management strategy (such as first-in-first-out) requires the buffer space to be recycled. If the workload exhibits reasonable spatial locality, the effect of pre-reading data can be achieved, because the workload often requests the remaining compressed data in a short period of time.

[0093] 4. After completing the data compression, update the mapping information of the FTL.

[0094] Since the compression operation changes the data size, the MLDC module needs to pass the compressed data size and the compression class to the FTL. The FTL saves the above information in the mapping table to correctly locate the stored data. The flash controller is responsible for writing the compressed data block to a continuous physical address.

[0095] The method of the above embodiment classifies page data into different compression categories based on machine learning technology without prompting of a software system, compresses and stores the same category using a shared dictionary, avoids repeated compression, realizes transparent and low-overhead data compression, and prolongs the service life of the solid state disk without increasing the cost of the device and the operation of the software. Moreover, although the data compression function is added in the present application, the MLDC module does not need the host system to provide additional information, so it can be compatible with the existing operating system and I / O interface without changing the host software stack, thereby providing a plug-and-play upgrade suitable for the existing system.

[0096] The data compression device provided by the present application is described below. The data compression device described below can be referred to in correspondence with the data compression method described above. The data compression device of the embodiment of the present application, as shown in Figure 5 includes:

[0097] The acquisition module 510 is configured to acquire data to be stored.

[0098] The determination module 520 is configured to determine a target compression dictionary according to the classification result of the data.

[0099] The compression module 530 is configured to compress the data according to the target compression dictionary to obtain compressed data.

[0100] Optionally, the determination module 520 is further configured to:

[0101] determine the classification result of the data according to data of a preset size in the data to be stored and a preset data classification model; and the data classification model is used to identify the type of the data to be stored.

[0102] Optionally, the compression module 530 is further configured to:

[0103] store the compression category information in a first flash translation layer (FTL) mapping table.

[0104] store the storage location of the compressed data and the size of the compressed data in a second FTL mapping table.

[0105] Optionally, the compression module 530 is further configured to:

[0106] if it is determined that the sum of the compressed data and the uncompressed data is less than a first value during the compression of the data to be stored, stop the compression of the uncompressed data and store the compressed data and the uncompressed data, in the case that the size of the data to be stored is greater than a first value and less than a second value; and the size of the first value is related to the size of the physical page of the solid state disk.

[0107] Optionally, the compression module 530 is further configured to:

[0108] In a case where the size of the data to be stored is greater than the second value, the threshold value of stopping compression corresponding to the data to be stored is determined according to the size of the sequential write data amount in the data to be stored, and the size of the random write data amount.

[0109] Optionally, the compression module 530 is further configured to:

[0110] The threshold value of stopping compression corresponding to the data to be stored is determined based on the following manner:

[0111]

[0112] wherein, represents the threshold value of stopping compression corresponding to the data to be stored; represents the size of the sequential write data amount in the data to be stored; r represents the size of the random write data amount in the data to be stored; p represents the size of a physical page in the solid state disk; and n represents a preset multiple value of the size of the physical page. and the compression dictionary size.

[0113] Figure 6 An example of an entity structure diagram of an electronic device is shown, which can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640. The processor 610, the communications interface 620, and the memory 630 can communicate with each other through the communications bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute a data compression method, which includes obtaining data to be stored, determining a target compression dictionary corresponding to the data according to a classification result of the data, and compressing the data according to the target compression dictionary to obtain compressed data.

[0114] In addition, the logical instruction in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0115] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the data compression method provided by the above-mentioned methods, which comprises: obtaining data to be stored; determining a target compression dictionary corresponding to the data according to the classification result of the data; and compressing the data according to the target compression dictionary to obtain compressed data.

[0116] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the data compression method provided by the above-mentioned methods, which comprises: obtaining data to be stored; determining a target compression dictionary corresponding to the data according to the classification result of the data; and compressing the data according to the target compression dictionary to obtain compressed data.

[0117] The device embodiments described above are only schematic, and the units described as separate units can or can not be physically separate, and the units displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0118] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0119] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data compression method characterized by, The method comprises: acquiring data to be stored; determining a target compression dictionary corresponding to the data according to a classification result of the data; compressing the data according to the target compression dictionary to obtain compressed data; The method further comprises: in a case where the size of the data to be stored is greater than a first value and less than a second value, if it is determined that the sum of the compressed data and uncompressed data is less than the first value during compression of the data to be stored, stopping compression of the uncompressed data and storing the compressed data and the uncompressed data; the size of the first value is related to the size of a physical page of a solid state disk; The method further comprises: in a case where the size of the data to be stored is greater than the second value, determining a threshold value for stopping compression corresponding to the data to be stored according to the size of sequential write data and the size of random write data in the data to be stored; the determination of the threshold value for stopping compression corresponding to the data to be stored according to the size of sequential write data and the size of random write data in the data to be stored comprises: determining the threshold value for stopping compression corresponding to the data to be stored based on the following manner: ； wherein, represents a threshold value of stopping compression corresponding to the data to be stored; represents a size of sequential write data amount in the data to be stored; r represents a size of random write data amount in the data to be stored; p represents a size of a physical page in a solid state disk; and n represents a preset multiple value of the size of the physical page; related to the size of the compression dictionary.

2. The data compression method of claim 1, wherein, The method further comprises: determining the classification result of the data according to data of a preset size in the data to be stored and a preset data classification model; the data classification model is used to identify the type of the data to be stored.

3. The data compression method of claim 1, wherein, The method further comprises: storing compression category information in a first flash translation layer (FTL) mapping table; storing the storage location of compressed data and the size of the compressed data in a second FTL mapping table.

4. A data compression device, characterized by, The method comprises: an acquiring module configured to acquire data to be stored; a determining module configured to determine a target compression dictionary corresponding to the data according to a classification result of the data; a compression module configured to compress the data according to the target compression dictionary to obtain compressed data; The compression module is further configured to, in a case where the size of the data to be stored is greater than a first value and less than a second value, if it is determined that the sum of the compressed data and uncompressed data is less than the first value during compression of the data to be stored, stop compression of the uncompressed data and store the compressed data and the uncompressed data; the size of the first value is related to the size of a physical page of a solid state disk. in a case where the size of the data to be stored is greater than the second value, determining a threshold value for stopping compression corresponding to the data to be stored according to the size of sequential write data and the size of random write data in the data to be stored; the determination of the threshold value for stopping compression corresponding to the data to be stored according to the size of sequential write data and the size of random write data in the data to be stored comprises: determining the threshold value for stopping compression corresponding to the data to be stored based on the following manner: ； wherein, represents a threshold value of stopping compression corresponding to the data to be stored; represents a size of sequential write data amount in the data to be stored; r represents a size of random write data amount in the data to be stored; p represents a size of a physical page in a solid state disk; and n represents a preset multiple value of the size of the physical page; related to the size of the compression dictionary.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the data compression method of any one of claims 1 to 3 when executing the program.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the data compression method of any one of claims 1 to 3 when executed by the processor.

7. A computer program product comprising a computer program, characterized in that, The computer program implements the data compression method of any one of claims 1 to 3 when executed by the processor.

Images