Data processing method and device, equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, equipment, a readable storage medium and a program product. The method comprises the following steps: determining to-be-compressed data; according to the to-be-compressed data, a target dictionary corresponding to the to-be-compressed data is determined, and the target dictionary is determined in advance according to the to-be-compressed data; and performing compression processing on the to-be-compressed data by using the target dictionary to obtain compressed data. By adopting the method, the compression rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, readable storage medium and program product. Background Art

[0002] Data compression technology is widely used in various fields. In order to improve the compression rate and compression speed, a compression algorithm with a dictionary is usually used.

[0003] Typically, a portion of feature data is selected through sampling to train a dictionary, and then the dictionary is used to compress / decompress all data with this feature data.

[0004] However, the above method has the problem of low compression rate. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, equipment, readable storage medium and program product that can improve the compression rate in response to the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, the method comprising:

[0007] Determine the data to be compressed;

[0008] Determine, according to the data to be compressed, a target dictionary corresponding to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0009] The target dictionary is used to compress the data to be compressed to obtain compressed data.

[0010] In this embodiment, a target dictionary is trained in advance using the data to be compressed, and when the data to be compressed is compressed, a target dictionary "dedicated" to the data to be compressed is selected for compression processing. This can avoid the problem of low compression rate caused by significant differences in data distribution between the training data of the target dictionary and the data to be compressed, thereby improving the compression rate of the data to be compressed.

[0011] In one embodiment, determining a target dictionary corresponding to the data to be compressed according to the data to be compressed includes:

[0012] According to the data to be compressed, obtaining a first Key corresponding to the data to be compressed;

[0013] If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

[0014] In this embodiment, a mapping relationship between data and a dictionary is established through key-value pairs. During the data compression process, it is convenient to try to hit the target Value corresponding to the first Key according to the first Key corresponding to the data to be compressed, so as to quickly find the target dictionary corresponding to the data to be compressed, which is conducive to improving compression efficiency.

[0015] In one embodiment, obtaining a first Key corresponding to the data to be compressed according to the data to be compressed includes:

[0016] Acquire a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed;

[0017] The first identifier, the first offset position, and the first data block quantity are used as the first Key.

[0018] In this embodiment, a triple formed by storing the first identifier of the first logical unit of the data to be compressed, the first offset position of the data to be compressed in the first logical unit, and the first number of data blocks corresponding to the data to be compressed represents the first Key corresponding to the data to be compressed. The method of determining the first Key does not require the server to perform additional calculations, which is conducive to saving computing resources.

[0019] In one of the embodiments, the target Value includes a dictionary identifier of the target dictionary.

[0020] In this embodiment, when establishing a key-value pair, a unique dictionary identifier is assigned to the dictionary as the target Value in the key-value pair. The implementation is simple and easy to implement, and does not occupy too much computing resources.

[0021] In one embodiment, before determining the data to be compressed, the method further includes:

[0022] Receive a data write request, and store the write data corresponding to the data write request in a disk;

[0023] The step of determining the data to be compressed includes:

[0024] The data to be compressed is determined from the written data stored in the disk.

[0025] In this embodiment, data compression is not involved in the data writing process, and the dictionary training and compression tasks are handled by the background thread, which can respond to data writing requests more quickly and improve the data writing experience.

[0026] In one embodiment, determining the data to be compressed from the written data stored in the disk includes:

[0027] According to the compression strategy, the written data stored in the first storage location in the disk is determined as the data to be compressed, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a writing time sequence of each storage location, and a compression mark of the written data.

[0028] In this embodiment, the compression strategy can be flexibly set during implementation, thereby improving compression flexibility.

[0029] In one embodiment, compressing the data to be compressed using the target dictionary to obtain compressed data includes:

[0030] Loading the data to be compressed from the disk into the memory, and loading the target dictionary from the disk into the memory;

[0031] The target dictionary is used to compress the data to be compressed in the memory to obtain the compressed data.

[0032] In this embodiment, when the target dictionary is successfully found in the disk, the data to be compressed and the target dictionary are loaded into the memory, and the data to be compressed is compressed in the memory, thereby reducing invalid memory usage.

[0033] In one embodiment, before determining the data to be compressed, the method further includes:

[0034] Determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to each target training data, wherein each target training data has the same number of data blocks as the data to be compressed;

[0035] For each of the dictionaries, a Key is generated according to the target training data, a Value corresponding to the dictionary is determined, and a key-value pair between the Key and the Value is established using a preset key-value storage library, each of the key-value pairs including the first key-value pair.

[0036] In this embodiment, a mapping relationship (key-value pair) between each Key and each Value is established based on the key-value storage library RocksDB. The key-value pairs are gradually sunk from the memory table to the SST on the disk and become part of the persistent storage. Such a hierarchical structure can ensure efficient management and fast reading of key-value pairs.

[0037] In one embodiment, determining the dictionary corresponding to each target training data according to each target training data includes:

[0038] Detecting whether there is historical training data whose similarity to the target training data is greater than a preset threshold;

[0039] If the historical training data exists, determining the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data;

[0040] If the historical training data does not exist, the dictionary is obtained by training according to the target training data.

[0041] In this embodiment, if there is historical training data that is identical to or highly similar to the target training data (i.e., the degree of similarity between the historical training data and the target training data is greater than a preset threshold), the dictionary corresponding to the historical training data is directly used as the dictionary corresponding to the target training data, thereby saving time for dictionary training and avoiding the occupation of storage space by duplicate dictionaries, thereby saving storage space occupied by dictionaries.

[0042] In one embodiment, the step of training the dictionary according to the target training data includes:

[0043] Preprocessing the target training data to obtain preprocessed data;

[0044] Performing frequency statistics on the preprocessed data to obtain frequency statistics results;

[0045] A high-frequency vocabulary dictionary is constructed according to the frequency statistics result, and the high-frequency vocabulary dictionary is used as the dictionary corresponding to the target training data.

[0046] In this embodiment, the preprocessing stage ensures the integrity and validity of the target training data. By recording high-frequency words, the efficiency can be improved in subsequent data compression. The generated dictionary set can bring extremely high compression rate and compression speed for these high-frequency words.

[0047] In one embodiment, the method further comprises:

[0048] Each of the key-value pairs is written into a write-ahead log.

[0049] Since the WAL log is a sequentially written log mechanism, in this embodiment, when a server failure occurs, the consistency of the key-value pair can be restored through the WAL log.

[0050] In one embodiment, the method further comprises:

[0051] The compressed data is stored in a second storage location in the disk, and a data access path corresponding to the compressed data is established according to the second storage location to replace the data access path corresponding to the data to be compressed.

[0052] In this embodiment, after the data to be compressed is compressed and the compressed data is obtained, the upper layer logical mapping is updated to ensure that the upper layer can access the storage location of the compressed data in the disk through the logical address.

[0053] In one embodiment, after the target dictionary is used to compress the data to be compressed to obtain compressed data, the method further includes:

[0054] A second Key is generated according to the compressed data, and the second Key is used to replace the first Key in the first key-value pair to obtain a second key-value pair.

[0055] Since the data to be compressed is compressed to obtain compressed data, the compressed data can be directly accessed later, and the first key-value pair is updated according to the compressed data to obtain the second key-value pair, which facilitates the subsequent rapid reading of the compressed data.

[0056] In one embodiment, the method further comprises:

[0057] The data to be compressed is marked as garbage data, and during a garbage collection process, garbage collection is performed on the garbage data.

[0058] In this embodiment, the target dictionary is used to compress the data to be compressed. After the compressed data is obtained, the data to be compressed can be marked as garbage data, and garbage collection can be performed on the garbage data during the garbage collection process, which can save disk storage space.

[0059] In one embodiment, the method further comprises:

[0060] When the compressed data needs to be decompressed, obtaining the second Key corresponding to the compressed data according to the compressed data;

[0061] According to the second Key, determine the target Value corresponding to the second Key in the second key-value pair;

[0062] The compressed data is decompressed using the target dictionary corresponding to the target Value.

[0063] In this embodiment, when the server decompresses the compressed data, it can quickly locate the target dictionary based on the second key-value pair, thereby improving the decompression rate.

[0064] In one embodiment, after decompressing the compressed data using the target dictionary corresponding to the target Value, the method further includes:

[0065] The usage frequency count value of the target dictionary is accumulated by 1;

[0066] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0067] This embodiment helps to accurately manage the storage and recycling of each dictionary by counting the usage frequency of each dictionary in detail; and as the compressed data increases, the dictionary of the embodiment of the present application will not expand indefinitely, and the server can recycle low-frequency dictionaries according to the usage frequency of each dictionary, and achieve the maximum data compression ratio within the smallest possible dictionary space. The dictionary recycling mechanism can optimize the occupancy of storage space, ensure that system resources are used efficiently, and improve the overall performance and response speed of the system, and prevent the storage space from being occupied by data that is not used for a long time. This is particularly important for high-concurrency and large-data volume application scenarios, and can avoid performance bottlenecks and system crashes caused by insufficient storage space.

[0068] In a second aspect, the present application provides a data processing method, the method comprising:

[0069] Determine the compressed data;

[0070] Determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data;

[0071] The compressed data is decompressed using the target dictionary.

[0072] In this embodiment, the target dictionary is trained in advance using the data to be compressed, and when the compressed data corresponding to the data to be compressed is decompressed, a "dedicated" target dictionary is selected to perform the decompression process, thereby improving the decompression efficiency.

[0073] In one embodiment, determining, based on the compressed data, a target dictionary corresponding to the compressed data includes:

[0074] According to the compressed data, obtaining a second Key corresponding to the compressed data;

[0075] According to the second Key, a target Value corresponding to the second Key is determined in a second key-value pair, and a dictionary corresponding to the target Value is used as the target dictionary.

[0076] In this embodiment, when the server decompresses the compressed data, it can quickly locate the target dictionary based on the second key-value pair, thereby improving the decompression rate.

[0077] In one embodiment, obtaining a second Key corresponding to the compressed data according to the compressed data includes:

[0078] Acquire a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data;

[0079] The second identifier, the second offset position, and the second data block quantity are used as the second Key.

[0080] In this embodiment, a triple formed by the second identifier of the second logical unit storing the compressed data, the second offset position of the compressed data in the second logical unit, and the second number of data blocks corresponding to the compressed data represents the second Key corresponding to the compressed data. The method for determining the second Key does not require the server to perform additional calculations, which is conducive to saving computing resources.

[0081] In one of the embodiments, the target Value includes a dictionary identifier of the target dictionary.

[0082] In this embodiment, when establishing a key-value pair, a unique dictionary identifier is assigned to the dictionary as the target Value in the key-value pair. The implementation is simple and easy to implement, and does not occupy too much computing resources.

[0083] In one embodiment, after decompressing the compressed data using the target dictionary, the method further includes:

[0084] The usage frequency count value of the target dictionary is accumulated by 1;

[0085] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0086] This embodiment helps to accurately manage the storage and recycling of each dictionary by counting the usage frequency of each dictionary in detail; and as the compressed data increases, the dictionary of the embodiment of the present application will not expand indefinitely, and the server can recycle low-frequency dictionaries according to the usage frequency of each dictionary, and achieve the maximum data compression ratio within the smallest possible dictionary space. The dictionary recycling mechanism can optimize the occupancy of storage space, ensure that system resources are used efficiently, and improve the overall performance and response speed of the system, and prevent the storage space from being occupied by data that is not used for a long time. This is particularly important for high-concurrency and large-data volume application scenarios, and can avoid performance bottlenecks and system crashes caused by insufficient storage space.

[0087] In a third aspect, the present application provides a data processing device, the device comprising:

[0088] A first determining module, used to determine the data to be compressed;

[0089] A second determination module, configured to determine a target dictionary corresponding to the data to be compressed according to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0090] The compression module is used to compress the data to be compressed using the target dictionary to obtain compressed data.

[0091] In a fourth aspect, the present application provides a data processing device, the device comprising:

[0092] A first determining module, used to determine compressed data;

[0093] A second determination module, configured to determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data;

[0094] A decompression module is used to decompress the compressed data using the target dictionary.

[0095] In a fifth aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect or the second aspect are implemented.

[0096] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect above are implemented.

[0097] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect above.

[0098] The above-mentioned data processing method, device, equipment, readable storage medium and program product determine the data to be compressed, and determine the target dictionary corresponding to the data to be compressed according to the data to be compressed. The target dictionary is determined in advance according to the data to be compressed, and then the data to be compressed is compressed using the target dictionary to obtain compressed data. In this way, due to the method of sampling and selecting a part of feature data to train the dictionary in the related art (the sampling method is, for example, random sampling), if there is a significant difference in data distribution between the selected training data and the actual data to be compressed, then the trained dictionary may not be able to effectively compress the actual data to be compressed, which will result in a low compression rate. The embodiment of the present application uses the data to be compressed to determine the target dictionary in advance. For example, the target dictionary can be trained in advance using the data to be compressed, and when the data to be compressed is compressed, a target dictionary "dedicated" to the data to be compressed is selected for compression processing, thereby avoiding the problem of low compression rate caused by the significant difference in data distribution between the training data of the target dictionary and the data to be compressed. The embodiment of the present application improves the compression rate of the data to be compressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0100] FIG1 is an application environment diagram of a data processing method in one embodiment;

[0101] FIG2 is a schematic flow chart of a data processing method in one embodiment;

[0102] FIG3 is a schematic diagram of a flow chart of step 202 in another embodiment;

[0103] FIG4 is a schematic flow chart of a data processing method in another embodiment;

[0104] FIG5 is a schematic diagram of a flow chart of step 203 in another embodiment;

[0105] FIG6 is a module diagram of a data processing method in another embodiment;

[0106] FIG7 is a schematic flow chart of a data processing method in another embodiment;

[0107] FIG8 is a schematic diagram of a module for training a dictionary in another embodiment;

[0108] FIG9 is a schematic diagram of a module for establishing a key-value pair in another embodiment;

[0109] FIG10 is a schematic flow chart of a data processing method in another embodiment;

[0110] FIG11 is a schematic flow chart of a data processing method in another embodiment;

[0111] FIG12 is a schematic flow chart of a data processing method in another embodiment;

[0112] FIG13 is a flow chart of a data processing method in another embodiment;

[0113] FIG14 is a module block diagram of a data writing core, a pre-training core, and a compression / decompression core in a processor in another embodiment;

[0114] FIG15 is a block diagram of a data processing device in one embodiment;

[0115] FIG16 is a block diagram of a data processing device in another embodiment;

[0116] FIG17 is a diagram showing the internal structure of a computer device in one embodiment;

[0117] FIG. 18 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0118] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0119] Data compression technology, as a means to effectively reduce the amount of data, is widely used in various fields, such as network communication, file transfer, data management, etc.

[0120] In order to improve the compression rate and compression speed, a compression algorithm with a dictionary is usually used. The compression algorithm with a dictionary creates a dictionary containing common data patterns (key-value pairs) and replaces the data fragments in the data to be compressed based on the data pattern during the compression process, thereby achieving data compression. Common compression algorithms with dictionaries include the LZ77 algorithm, the LZ78 algorithm, the LZMA (Lempel-Ziv-Markov chain algorithm) algorithm, the ZSTD (Zstandard) algorithm, the Brotli algorithm, and the LZ4 algorithm. In these compression algorithms with dictionaries, the dictionaries can be static or dynamically generated, and the way they maintain dictionaries is slightly different.

[0121] For example, it is common to select a portion of feature data from a data set through sampling to train a dictionary, and then use the dictionary to compress / decompress all data with this feature data. Sampling methods are divided into random sampling, stratified sampling, and time sampling. Random sampling is to randomly extract a portion of data from the entire data set. Stratified sampling is to perform stratified sampling based on the category or feature of the data to ensure that each type of data is represented. Time sampling means that for time series data, sampling can be performed from different time periods to ensure that the time characteristics of the data are preserved.

[0122] However, the sampled training data may not fully represent the characteristics of all the data to be compressed. If certain specific data are selected intentionally or unintentionally during the sampling process, and there is a significant difference in the data distribution between the selected training data and the actual data to be compressed, then the generated dictionary will also be biased towards these training data and will not be able to effectively compress all the actual data to be compressed, resulting in a low compression rate.

[0123] In view of this, embodiments of the present application provide a data processing method, apparatus, device, readable storage medium, and program product, which can improve the compression rate of data to be compressed.

[0124] The data processing method provided in the embodiment of the present application can be applied in the implementation environment shown in FIG1 , which includes a server, which can be implemented as an independent server or a server cluster composed of multiple servers, and a data storage system can store data that the server needs to process. The data storage system can be integrated on the server, or placed on the cloud or other network servers.

[0125] In other possible implementations, the data processing method provided in the embodiments of the present application may also be applied to terminals, which may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0126] In an exemplary embodiment, as shown in FIG2 , a data processing method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps 201 to 203. Among them:

[0127] Step 201: determine the data to be compressed.

[0128] In the embodiment of the present application, after receiving a data write request, the server directly writes the write data corresponding to the data write request into the disk to form a data block, and when compression processing is required later, the data to be compressed is determined from the disk.

[0129] Optionally, the server may sequentially determine a preset number of data blocks in the disk as data to be compressed, and the preset number may be set at the time of implementation.

[0130] Optionally, the server may also write the same data into one or more data blocks corresponding to the request in the disk as data to be compressed.

[0131] Optionally, the server may also actively select one or more data blocks in the disk as data to be compressed, and so on.

[0132] Step 202: determining a target dictionary corresponding to the data to be compressed according to the data to be compressed. The target dictionary is determined in advance according to the data to be compressed.

[0133] After the server writes the write data directly to the disk, the server determines the dictionary based on the write data. Optionally, the server can pre-train the dictionary based on the write data and store the trained dictionary. The way in which the server pre-trains the dictionary will be described in the following embodiments. Optionally, the server can also determine the dictionary corresponding to the data that is the same as or highly similar to the write data as the dictionary of the write data.

[0134] In this way, after the server determines the data to be compressed, the server searches for the target dictionary corresponding to the data to be compressed. It is understandable that the target dictionary may be pre-trained based on the data to be compressed, or the target dictionary may be pre-trained based on data that is the same as or highly similar to the data to be compressed.

[0135] Exemplarily, the server's disk can also be divided into a data storage area and a dictionary storage area. The data storage area stores write data, and the dictionary storage area stores various pre-trained dictionaries. The server can determine the data to be compressed in the data storage area, and search in the dictionary storage area whether there is a target dictionary corresponding to the data to be compressed.

[0136] Taking the example that the target dictionary is trained based on the data to be compressed, as an implementation method, during the pre-training dictionary stage, if the server has trained the target dictionary based on the data to be compressed, the server can add the same identification field to the data to be compressed and the target dictionary. In this way, in step 202, the server can search for the target dictionary with the identification field in the dictionary storage area based on the identification field of the data to be compressed.

[0137] Other possible implementations of step 202 will be introduced in the following embodiments.

[0138] Step 203: compress the data to be compressed using the target dictionary to obtain compressed data.

[0139] If the server finds the target dictionary corresponding to the data to be compressed according to the data to be compressed, the server will pass the data to be compressed and the target dictionary together into the compression algorithm with the dictionary to perform data compression and obtain compressed data.

[0140] Exemplarily, continuing with the example of the server adding the same identification field to the data to be compressed and the target dictionary in the pre-training dictionary stage, after step 203, the server can also add the identification field to the compressed data. In this way, in the decompression stage of the compressed data, after the server determines the compressed data, it can determine the target dictionary corresponding to the compressed data based on the compressed data. For example, the server can search for the target dictionary with the identification field in the dictionary storage area based on the identification field of the compressed data, and then the server uses the target dictionary to decompress the compressed data to obtain the original data, that is, the above-mentioned data to be compressed.

[0141] It should be noted that if the server fails to find the target dictionary corresponding to the data to be compressed, it can give up the compression process of the data to be compressed to avoid unnecessary processing overhead, and try to find the target dictionary again when the next background compression task is initiated; or, the server can also use other compression algorithms without dictionaries to compress the data to be compressed, and so on.

[0142] In one possible implementation, the processor of the server can be divided into multiple cores, for example, a data writing core, a pre-training core, and a compression / decompression core, wherein the data writing core is used to write data to the disk, and it belongs to the front-end core of the processor, the pre-training core is used for pre-training of the dictionary, and it belongs to the back-end core of the processor, and the compression / decompression core is used for data compression and data decompression with a dictionary, and it belongs to the back-end core of the processor. The server can use the compression / decompression core to compress the compressed data using the target dictionary to obtain compressed data.

[0143] The above-mentioned embodiment uses the data to be compressed to determine the target dictionary in advance, and selects the target dictionary "dedicated" to the data to be compressed for compression processing when compressing the data to be compressed, thereby avoiding the problem of low compression rate caused by significant differences in data distribution between the training data of the target dictionary and the data to be compressed. The embodiment of the present application improves the compression rate of the data to be compressed.

[0144] In one embodiment, based on the embodiment shown in FIG2, referring to FIG3, this embodiment shows another implementation method in which the server determines the target dictionary corresponding to the data to be compressed according to the data to be compressed. As shown in FIG3, step 202 includes steps 301 and 302 shown in FIG3:

[0145] Step 301: Obtain a first Key corresponding to the data to be compressed according to the data to be compressed.

[0146] In the embodiment of the present application, during the pre-training dictionary stage, after the server obtains different dictionaries according to different training data, the server can establish a KV (Key-target Value) mapping relationship between the training data and the dictionary, that is, a key-value pair, and the server records the key-value pair in a table structure on the disk. Optionally, for multiple training data that are the same or highly similar, a dictionary can be obtained by training only one of the training data, and the multiple training data can share the dictionary, so that the dictionary and the multiple training data respectively establish corresponding key-value pairs.

[0147] In this way, after determining the data to be compressed in step 201, the server can obtain the first Key corresponding to the data to be compressed according to the data to be compressed, and search the target dictionary according to the first Key.

[0148] After the server determines the data to be compressed, in a possible implementation, the server can obtain the storage location of the data to be compressed on the disk and the number of data blocks included in the data to be stored (for the sake of ease of description, referred to here as the first number of data blocks), and obtain the first Key corresponding to the data to be compressed based on the storage location and the first number of data blocks.

[0149] Exemplarily, the server can obtain the first identifier of the first logical unit storing the data to be compressed, the first offset position of the data to be compressed in the first logical unit, and the first number of data blocks corresponding to the data to be compressed, that is, the storage location of the data to be compressed in the disk includes the first identifier of the first logical unit storing the data to be compressed and the first offset position of the data to be compressed in the first logical unit, and then, the server uses the first identifier, the first offset position and the first number of data blocks as the first Key.

[0150] The first identifier may be the number of the first logical unit. Assume that lun is used to represent the logical unit of disk partition, lunId is the number of the logical unit, lba represents the offset position of the data on a certain logical unit, and blockNr represents the total number of data blocks (blocks). Exemplarily, the amount of data in a block may be 8192 Bytes. Therefore, in this embodiment, Key is a triple formed by lunId, lba, and blockNr.

[0151] In this way, the triple formed by storing the first identifier of the first logical unit of the data to be compressed, the first offset position of the data to be compressed in the first logical unit, and the first number of data blocks corresponding to the data to be compressed represents the first Key corresponding to the data to be compressed. The method of determining the first Key does not require the server to perform additional calculations, which is conducive to saving computing resources.

[0152] In another possible implementation, the server may also use the first identifier of the first logical unit storing the data to be compressed and the first offset position of the data to be compressed in the first logical unit as the first Key corresponding to the data to be compressed, and so on. The form of the first Key is not specifically limited here.

[0153] Step 302: If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

[0154] After the server obtains the first Key corresponding to the data to be compressed through the implementation of step 301, it attempts to try whether it can hit the target Value corresponding to the first Key in each key-value pair according to the first Key. It can be understood that if each key-value pair includes the first key-value pair corresponding to the first Key, then the server can determine the target Value corresponding to the first Key in the first key-value pair according to the first Key.

[0155] After the server determines the target Value corresponding to the first Key, the server uses the dictionary represented by the target Value as the target dictionary.

[0156] Optionally, after the server obtains the dictionary through training, it can assign a dictionary identifier to the dictionary. The dictionary identifier can be, for example, a dictionary number. The dictionary number of each dictionary is not repeated. The dictionary number is used as the target Value. In this way, after the server determines the target Value corresponding to the first Key, the target Value includes the dictionary identifier of the target dictionary (such as the dictionary number of the target dictionary). The server determines the target dictionary through the dictionary identifier. In this implementation, when establishing a key-value pair, a unique dictionary identifier is assigned to the dictionary as the target Value in the key-value pair. The implementation is simple and easy to implement, and does not occupy too much computing resources.

[0157] Optionally, after the server obtains the dictionary through training, it can also use the storage location of the dictionary on disk as the target Value to create a key-value pair, and so on.

[0158] In the above embodiment, a mapping relationship between data and a dictionary is established through key-value pairs. During data compression, the target Value corresponding to the first Key can be easily tried according to the first Key corresponding to the data to be compressed, so that the target dictionary corresponding to the data to be compressed can be quickly found, which is conducive to improving compression efficiency.

[0159] In one embodiment, based on any of the above embodiments, taking the embodiment shown in FIG. 2 as an example, referring to FIG. 4 , in the data processing method of this embodiment, before step 201, the method further includes step 401 shown in FIG. 4 :

[0160] Step 401, receiving a data write request, and storing write data corresponding to the data write request in a disk.

[0161] The server receives a data write request, which includes write data. The write data is passed to the storage driver through the system call interface of the operating system. The storage driver passes the write data to the storage array, reorganizes the write data in the storage array, and finally writes it to the disk.

[0162] Correspondingly, please continue to refer to FIG. 4 , step 201 may include step 2011 shown in FIG. 4 :

[0163] Step 2011, determining the data to be compressed from the write data stored in the disk.

[0164] The data to be compressed may include write data corresponding to multiple data write requests, the data to be compressed may include write data corresponding to one data write request, the data to be compressed may also include partial data in the write data corresponding to one or more data write requests, and so on. No specific limitation is made here on the correspondence between write data and data write requests.

[0165] This embodiment does not involve data compression during the data writing process, and the dictionary training and compression tasks are handed over to the background thread for processing, which can respond to data writing requests more quickly and improve the data writing experience.

[0166] In a possible implementation of step 2011, the server may determine the data stored in the first storage location in the disk as data to be compressed according to the compression strategy.

[0167] After receiving the data write request, the server writes the write data directly to the disk to form a data block. When compression processing is required later, the server determines the data to be compressed from the write data on the disk according to the compression strategy.

[0168] The compression strategy includes at least one of a preset number of data blocks, a preset data volume, a writing time sequence of each storage location, and a compression mark of the written data.

[0169] After the written data is written to the disk, it is stored in the disk in blocks. For example, the data volume of each block is 8192Byte. Therefore, optionally, when compressing the data, the server can select the data to be compressed from the written data in the disk according to the preset number of data blocks, wherein the number of data blocks corresponding to the data to be compressed is consistent with the number of data blocks selected by the server for the target training data in the pre-training dictionary stage. For example, when the server trains the dictionary, it selects 512 data blocks to train a dictionary, that is, the target training data is 512 data blocks. Then, when compressing the data, the server also determines 512 data blocks as the data to be compressed each time, that is, the target training data and the data to be compressed have the same number of data blocks, so that the uniqueness of the Key can be maintained, and the storage location of the 512 data blocks in the disk is called the first storage location, and the server determines the data stored in the first storage location in the disk as the data to be compressed.

[0170] Optionally, the server may also select the data to be compressed this time from the written data in the disk according to a preset data volume; optionally, the server may also select the data to be compressed this time from the written data in the disk according to the chronological order of the writing time of each storage location. For example, the server may determine the data to be compressed in order from the earliest to the latest writing time; optionally, the server may also select the data to be compressed this time from the written data in the disk according to the compression identifier, as long as the number of data blocks corresponding to the data to be compressed is ensured to be consistent with the number of data blocks of the target training data selected by the server in the pre-training dictionary stage. For example, the server may determine the data stored in the first storage location as the data to be compressed when the data stored in the first storage location has a compression identifier, that is, the data to be compressed may be flexibly selected through the compression identifier. The compression identifier may be added by the server according to a preset rule or may be manually set. When the server performs data compression, it selectively compresses the data with the compression identifier, thereby improving the compression flexibility.

[0171] In this embodiment, the compression strategy can be flexibly set during implementation, which improves the flexibility of compression. In this embodiment of the application, by using a target dictionary pre-trained in the background to compress the data to be compressed, it can save foreground resources and does not affect the foreground data writing performance. At the same time, due to the existence of the dictionary, the compression rate and compression speed of the background are improved.

[0172] In one embodiment, based on the embodiment shown in FIG4, referring to FIG5, this embodiment exemplarily introduces the process of the server using the target dictionary to compress the data to be compressed to obtain the compressed data. As shown in FIG5, step 203 may include steps 501 and 502 shown in FIG5:

[0173] Step 501: load the data to be compressed from the disk into the memory, and load the target dictionary from the disk into the memory.

[0174] Step 502: compress the data to be compressed using the target dictionary in the memory to obtain compressed data.

[0175] In an embodiment of the present application, after the server determines the data to be compressed on the disk, it does not directly load the data to be compressed into the memory, but continues to search the disk for a target dictionary corresponding to the data to be compressed. If the target dictionary is successfully found, the server loads the data to be compressed and the target dictionary into the memory, and passes the target dictionary and the data to be compressed into the compression algorithm with the dictionary in the memory, compresses the data to be compressed, and obtains compressed data.

[0176] The following is an example description of the data compression process of the server in conjunction with FIG6 .

[0177] As described above, the server obtains the first Key based on the data to be compressed, that is, the triplet (lunId, lba, blockNr) corresponding to the data to be compressed, and tries to hit the target Value based on the triplet. After hitting the target Value (including the dictionary identifier of the target dictionary), the server reads the data to be compressed and the corresponding target dictionary from the disk respectively, and passes the target dictionary and the data to be compressed into the compression algorithm with dictionary in the memory. The data to be compressed is compressed according to each data block. Compression algorithms with dictionaries such as LZ4 and ZSTD can use pre-trained target dictionaries to improve the compression rate and compression speed of the data to be compressed.

[0178] In the above embodiment, when the server successfully finds the target dictionary in the disk, the server loads the data to be compressed and the target dictionary into the memory, and compresses the data to be compressed in the memory, thereby reducing invalid memory usage.

[0179] Furthermore, the server compresses the data to be compressed, and after obtaining the compressed data, the server can also store the compressed data in a second storage location on the disk. Since the data to be compressed is written to the second storage location on the disk after being compressed, the data to be compressed becomes garbage data. The server establishes a data access path corresponding to the compressed data according to the second storage location to replace the data access path corresponding to the data to be compressed. That is, the server compresses the data to be compressed, and after obtaining the compressed data, updates the upper-level logical mapping to ensure that the upper layer can access the storage location of the compressed data on the disk through the logical address.

[0180] In one embodiment, based on the embodiment shown in FIG3 , the process of determining the dictionary by the server is described below as an example. As shown in FIG7 , before the server determines the data to be compressed, the data processing method of this embodiment further includes steps 701 and 702 shown in FIG7 :

[0181] Step 701 : determining a plurality of target training data from the written data stored in the disk, and determining a dictionary corresponding to each target training data according to the target training data.

[0182] Among them, each target training data and the data to be compressed have the same number of data blocks.

[0183] As described above, after the server stores the write data corresponding to the data write request on the disk, the write data is stored on the disk in blocks, and each data block has a unique logical block address (LBA mentioned above) to describe its location.

[0184] In a possible implementation, the server may sequentially read a fixed number of data blocks from the disk into the memory (for example, the data size of each block is 8192 bytes, and the fixed number is, for example, 512, then 512 data blocks are equivalent to 4MB of data) as a target training data, and train a dictionary based on the target training data. Similarly, during the process of pre-training the dictionary, the server will train multiple dictionaries based on multiple target training data, that is, the server trains a dictionary for each target training data. The pre-training method may vary depending on the compression algorithm used.

[0185] It should be noted that in the embodiment of the present application, the number of data blocks when training the dictionary can be flexibly set. Different numbers of data blocks determine different training speeds and compression efficiencies. In the actual implementation process, the number of data blocks can be flexibly selected according to the required training speed and compression efficiency.

[0186] After the training is completed, the server stores the multiple dictionaries in the disk. Exemplarily, the server may store the multiple dictionaries in the above dictionary storage area in the disk.

[0187] In an embodiment of the present application, the server can periodically determine multiple target training data from the write data stored in the disk to perform dictionary training. The server can also determine multiple target training data from the write data stored in the disk to perform dictionary training when the amount of write data stored in the disk is greater than a threshold.

[0188] In another possible implementation, in order to avoid repeatedly training the dictionary for the same training data or training data with high similarity, which results in a waste of computing resources and storage resources, after the server determines the target training data, the server may also first detect whether there is historical training data whose similarity with the target training data is greater than a preset threshold. It should be noted that in the process of historical pre-training the dictionary, the server has already trained the corresponding dictionary based on the historical training data.

[0189] Exemplarily, the server may calculate the similarity between the target training data and the historical training data through a similarity algorithm. If the similarity is greater than a preset threshold, it indicates that the target training data and the historical training data have a high similarity. Exemplarily, the server may also calculate the first hash value of the target training data and the second hash value of the historical training data. If the first hash value is the same as the second hash value, it indicates that the target training data and the historical training data have a high similarity.

[0190] In this way, if historical training data exists, the server does not need to repeatedly train the dictionary for the target training data, but can directly determine the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data; if historical training data does not exist, the dictionary is trained based on the target training data.

[0191] In this embodiment, if there is historical training data that is identical to or highly similar to the target training data (i.e., the degree of similarity between the historical training data and the target training data is greater than a preset threshold), the dictionary corresponding to the historical training data is directly used as the dictionary corresponding to the target training data, thereby saving time for dictionary training and avoiding the occupation of storage space by duplicate dictionaries, thereby saving storage space occupied by dictionaries.

[0192] The following is an exemplary introduction to the process of the server training a dictionary based on target training data. The process may include the following steps A1 to A3:

[0193] Step A1, preprocessing the target training data to obtain preprocessed data.

[0194] In the preprocessing stage, the server first needs to clean up the zero data and truncated data in the target training data. Zero data refers to invalid data blocks with all bytes being zero, and truncated data refers to data that cannot be stored completely due to various reasons. The server can identify truncated data by identifying the truncation mark in the data block. After the server cleans up the zero data and truncated data in the target training data, it will standardize the remaining valid data to obtain the preprocessed data.

[0195] Step A2, performing frequency statistics on the preprocessed data to obtain frequency statistics results.

[0196] After the preprocessing is completed, the server performs frequency statistics on the preprocessed data to obtain high-frequency words in the target training data (i.e., frequency statistics results). These high-frequency words represent common patterns in the target training data and are the basis for building a dictionary.

[0197] Step A3: construct a high-frequency vocabulary dictionary based on the frequency statistics results, and use the high-frequency vocabulary dictionary as the dictionary corresponding to the target training data.

[0198] The server constructs a high-frequency vocabulary dictionary based on the frequency statistics results, thereby obtaining a dictionary "dedicated" to the target training data.

[0199] Exemplarily, in conjunction with FIG8 , the process of obtaining a dictionary by training the server according to target training data is illustrated.

[0200] Assume that the server reads 512 data blocks as a target training data. The server performs a series of processing on the target training data in sequence, including preprocessing, frequency statistics, and building a high-frequency vocabulary dictionary. Finally, multiple target training data can generate a dictionary set.

[0201] In this way, the preprocessing stage ensures the integrity and validity of the target training data. By recording high-frequency words, the efficiency can be improved in subsequent data compression. The generated dictionary set can bring extremely high compression rate and compression speed for these high-frequency words.

[0202] Step 702, for each dictionary, generate a Key according to the target training data, determine the Value corresponding to the dictionary, and use a preset key-value storage library to establish a key-value pair between the Key and the Value.

[0203] Among them, each key-value pair includes the above-mentioned first key-value pair.

[0204] As described above, the server can generate a triple (lunId, lba, blockNr) corresponding to the target training data as the Key, and use the dictionary identifier of the dictionary as the Value. For each dictionary, the server establishes a key-value pair between the Key and the Value according to the key-value storage library. In this way, when the server searches for the target dictionary corresponding to the data to be compressed, it can hit the first key-value pair in each established key-value pair according to the first Key, thereby finding the dictionary identifier of the target dictionary.

[0205] Exemplarily, in combination with Figure 9, in an embodiment of the present application, the server can establish a mapping relationship (key-value pair) between each Key and each Value based on the key-value storage library RocksDB, and the key-value pairs are gradually sunk from the memory table (MemTable) to the SST on the disk and become part of the persistent storage. Such a hierarchical structure can ensure efficient management and fast reading of key-value pairs.

[0206] When each key-value pair is generated, the server can also write each key-value pair into a write-ahead log (WAL log). The WAL log is a sequentially written log mechanism. When a server fails, the consistency of the key-value pair can be restored through the WAL log.

[0207] The above embodiments introduce the processes of the server performing compression processing on the data to be compressed, the server pre-training the dictionary, etc. The following is an exemplary introduction to the relevant processes of the server performing decompression processing.

[0208] In one embodiment, based on the embodiment shown in FIG. 3 , referring to FIG. 10 , the server in this embodiment uses the target dictionary to compress the data to be compressed. After obtaining the compressed data, the data processing method further includes step 1001:

[0209] Step 1001, generate a second key according to the compressed data, and use the second key to replace the first key in the first key-value pair to obtain a second key-value pair.

[0210] The way the server generates the second key based on the compressed data is similar to the way the server generates the first key based on the data to be compressed, which will not be repeated here. After the server generates the second key, it uses the second key to replace the first key in the first key-value pair to obtain the second key-value pair, and the first key-value pair is invalidated.

[0211] In this way, since the data to be compressed is compressed to obtain compressed data, the compressed data can be directly accessed later, and the first key-value pair is updated according to the compressed data to obtain the second key-value pair, which facilitates the subsequent rapid reading of the compressed data.

[0212] Furthermore, the server compresses the data to be compressed using the target dictionary. After obtaining the compressed data, the server may also mark the data to be compressed as garbage data and perform garbage collection on the garbage data during the garbage collection process, thereby saving disk storage space.

[0213] In one embodiment, based on the embodiment shown in FIG. 10 , referring to FIG. 11 , the data processing method further includes steps 1101 to 1103 shown in FIG. 11 :

[0214] Step 1101, when the compressed data needs to be decompressed, obtain the second Key corresponding to the compressed data according to the compressed data.

[0215] Exemplarily, if the upper layer needs to access compressed data, the server can decompress the compressed data. The server first obtains the second Key corresponding to the compressed data based on the compressed data, wherein the server obtains the second Key corresponding to the compressed data in the same way as the server obtains the first Key corresponding to the data to be compressed. For example, the server can obtain the second identifier of the second logical unit storing the compressed data, the second offset position of the compressed data in the second logical unit, and the second number of data blocks corresponding to the compressed data, and use the second identifier, the second offset position, and the second number of data blocks as the second Key.

[0216] Step 1102: Determine the target Value corresponding to the second Key in the second key-value pair according to the second Key.

[0217] The server attempts to hit the Value according to the second Key, and determines the target Value corresponding to the second Key in the second key-value pair. As described above, the target Value includes the dictionary identifier (such as the dictionary number) of the target dictionary.

[0218] Step 1103, decompress the compressed data using the target dictionary corresponding to the target Value.

[0219] After the server determines the target Value, it can determine the target dictionary corresponding to the target Value. The server loads the target dictionary and compressed data into memory, and decompresses the compressed data using the target dictionary in memory to obtain the original uncompressed data.

[0220] In this way, based on the second key-value pair, the server can also quickly locate the target dictionary when decompressing the compressed data, thereby improving the decompression rate.

[0221] Optionally, after the server uses the target dictionary corresponding to the target Value to decompress the compressed data, it may also add 1 to the usage frequency count value of the target dictionary.

[0222] For example, a frequency counter may be set for each dictionary. Whenever the dictionary is accessed, the value of the frequency counter may be increased. The frequency counter may help track the usage of each dictionary and identify those dictionaries that are not frequently accessed for recycling.

[0223] This embodiment helps to accurately manage the storage and recycling of each dictionary by counting the usage frequency of each dictionary in detail.

[0224] In this way, if the usage frequency count value is less than the preset threshold value within the preset time period, the target dictionary and the second key-value pair are deleted.

[0225] As the amount of compressed data increases, the dictionary in the embodiment of the present application will not expand indefinitely, and the server can recycle low-frequency dictionaries based on the frequency of use of each dictionary to achieve the maximum data compression ratio within the smallest possible dictionary space.

[0226] Exemplarily, the server may monitor the usage frequency count of each dictionary. If the usage frequency count is less than a preset threshold, the dictionary and its corresponding key-value pair are marked as recycling objects. Subsequently, the server automatically cleans up these recycling objects and releases the storage space they occupy.

[0227] This embodiment is based on the least frequency of use strategy. When the usage frequency of a dictionary is lower than a preset threshold, a recycling operation is performed to delete the key-value pairs corresponding to the dictionary and the dictionary itself, thereby controlling the size of the entire dictionary set. The dictionary is retrained when it is needed next time.

[0228] In this embodiment, the dictionary recycling mechanism can optimize the occupancy of storage space, ensure that system resources are used efficiently, and improve the overall performance and response speed of the system. It can prevent the storage space from being occupied by data that is not used for a long time. This is especially important for application scenarios with high concurrency and large data volumes, and can avoid performance bottlenecks and system crashes caused by insufficient storage space.

[0229] In one possible implementation, after the server deletes the target dictionary and the second key-value pair, if it receives an access request for compressed data corresponding to the target dictionary, since the target dictionary and the second key-value pair have been deleted, the server can decompress the compressed data in a dictionary-free manner to obtain the original data, and respond to the access request based on the original data to ensure a smooth response to the access request.

[0230] Then, the server may train a dictionary based on the original data. For the manner in which the server trains a dictionary based on the original data, reference may be made to the relevant description of the above embodiment, which will not be described in detail here.

[0231] After the server obtains a dictionary through training based on the original data, the server may establish a key-value pair between the dictionary and the compressed data according to the implementation method in the above embodiment, and store the key-value pair and the trained dictionary for subsequent use in decompressing the compressed data.

[0232] Optionally, the server may also add 1 to the usage frequency count value of the dictionary, and track the usage of the dictionary during subsequent usage of the dictionary to identify infrequently accessed dictionaries for recycling, thereby ensuring efficient use of system resources.

[0233] In one embodiment, as shown in FIG. 12 , a data processing method is provided, which is described by taking the method applied to the server in FIG. 1 as an example, and includes the following steps 1201 to 1203. Among them:

[0234] Step 1201, determine compressed data.

[0235] Step 1202: determine a target dictionary corresponding to the compressed data according to the compressed data. The target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data.

[0236] Step 1203: Decompress the compressed data using the target dictionary.

[0237] In one embodiment, determining a target dictionary corresponding to the compressed data according to the compressed data includes:

[0238] According to the compressed data, obtain the second Key corresponding to the compressed data;

[0239] According to the second Key, the target Value corresponding to the second Key is determined in the second key-value pair, and the dictionary corresponding to the target Value is used as the target dictionary.

[0240] In one embodiment, determining, according to the compressed data, a second Key corresponding to the compressed data includes:

[0241] Acquire a second identifier of a second logical unit storing compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data;

[0242] The second identifier, the second offset position and the second data block quantity are used as the second Key.

[0243] In one embodiment, the target Value includes a dictionary identifier of a target dictionary.

[0244] In one embodiment, after determining the target dictionary corresponding to the compressed data according to the compressed data, the method further includes:

[0245] The usage frequency count value of the target dictionary is accumulated by 1;

[0246] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0247] For the implementation and beneficial effects of the embodiment shown in FIG12 , please refer to the relevant description in the above embodiment, which will not be repeated here.

[0248] In one embodiment, as shown in FIG. 13 , a data processing method is provided, which is described by taking the method applied to the server in FIG. 1 as an example, and includes the following steps:

[0249] Step 1301, write data to disk and make it persistent.

[0250] After receiving the data write request, the server writes the write data corresponding to the data write request to the disk and persists it.

[0251] Step 1302, read multiple target training data from the disk, and train multiple dictionaries.

[0252] The server reads multiple target training data from the write data stored in the disk, and trains a dictionary according to each target training data to obtain multiple dictionaries. It should be noted that this embodiment is introduced by taking the example that the server trains each target training data to obtain a dictionary. Optionally, for multiple target training data that are the same or highly similar, a dictionary can be obtained by training only one of the target training data, and the multiple target training data can share the dictionary.

[0253] Step 1303: Create a key-value pair between each target training data and the corresponding dictionary, and write each dictionary and each key-value pair to disk for persistence.

[0254] For each dictionary, the server generates a key according to the target training data corresponding to the dictionary, generates a value according to the dictionary, and uses a preset key-value storage library to establish a key-value pair between the key and the value, and stores each dictionary and each key-value pair to the disk.

[0255] Step 1304, determine the data to be compressed from the disk, and obtain the first Key corresponding to the data to be compressed.

[0256] The server determines the data to be compressed from the written data stored in the disk, and obtains the first Key corresponding to the data to be compressed according to the data to be compressed.

[0257] Step 1305, search for the target Value corresponding to the first Key.

[0258] The server searches for the target value corresponding to the first Key in each key-value pair based on the first Key.

[0259] Step 1306: If the target Value is found, the compression algorithm with dictionary is used to compress the data to be compressed and make it persistent.

[0260] If the server determines the target Value corresponding to the first Key in the first key-value pair, it loads the data to be compressed from the disk to the memory, and loads the target dictionary from the disk to the memory. In the memory, the target dictionary and the data to be compressed are passed into the compression algorithm with the dictionary for compression processing to obtain compressed data.

[0261] Step 1307: Recover the data to be compressed, and recover the low-heat dictionary and its key-value pairs.

[0262] The low-heat dictionary may refer to a dictionary whose usage frequency within a preset time period is less than a preset threshold.

[0263] Step 1308: If the target Value is not found, the data compression process is abandoned.

[0264] In one embodiment, as described above, the processor of the server can be divided into multiple cores, for example, a data writing core, a pre-training core, and a compression / decompression core, wherein the data writing core is used to write data to the disk, and it belongs to the front-end core of the processor, the pre-training core is used for pre-training of the dictionary, and it belongs to the back-end core of the processor, and the compression / decompression core is used for data compression and data decompression with a dictionary, and it belongs to the back-end core of the processor. The server can use the compression / decompression core to compress the compressed data using the target dictionary to obtain compressed data. In the following, the data processing method of the embodiment of the present application is exemplarily described in combination with the data writing core, the pre-training core, and the compression / decompression core.

[0265] Please refer to Figure 14, which is a module block diagram of the data writing core, pre-training core, and compression / decompression core in the processor. It should be noted that Figure 14 only shows the processor core related to the embodiment of the present application, and other cores of the processor are not shown in Figure 14. The following steps are included:

[0266] 1. Write data to disk and persist it.

[0267] The data writing core writes the write data corresponding to the data writing request to the disk and persists it.

[0268] 2. The pre-training core reads multiple target training data from disk and pre-trains multiple dictionaries in the background.

[0269] The pre-training core reads a plurality of target training data from the write data stored in the disk, and trains a dictionary according to each target training data to obtain a plurality of dictionaries.

[0270] 3. The pre-training core creates a key-value pair for each target training data and the corresponding dictionary, and writes each dictionary and each key-value pair to disk for persistence.

[0271] For each dictionary, the pre-training core generates a key based on the target training data corresponding to the dictionary, generates a value based on the dictionary, and uses the preset key-value storage library to establish a key-value pair between the key and the value, and stores each dictionary and each key-value pair to the disk.

[0272] 4. The compression / decompression core reads data from the disk and finds the dictionary corresponding to the data for compression / decompression.

[0273] 5. The server recycles low-temperature dictionaries and their key-value pairs.

[0274] The embodiment of the present application compresses data by using a dictionary pre-trained in the background, which can save foreground resources without affecting the foreground data writing performance. At the same time, due to the existence of the dictionary, the compression rate and compression speed of the background are improved.

[0275] In the embodiment of the present application, different data corresponds to different dictionaries. As the amount of compressed data increases, the dictionary will not expand indefinitely. The server will recycle low-frequency dictionaries based on the frequency of use of the dictionary to achieve the maximum data compression ratio within the smallest possible dictionary space.

[0276] In some related technologies, there is a method of updating the dictionary by dynamically training input data, such as configuring replaceable entries in the dictionary, but frequent dictionary updates without timely cleaning of old dictionary entries will make the dictionary larger and larger, increasing the burden of storage and management. In the embodiment of the present application, different data correspond to different dictionaries, so it has the ability of high compression rate and high compression speed, and the timed recovery of the dictionary can optimize the occupancy of storage space, ensure that system resources are efficiently used, and improve the overall performance and response speed of the system, which can prevent the storage space from being occupied by data that is not used for a long time. This is particularly important for high concurrency and large data volume application scenarios, and can avoid performance bottlenecks and system crashes caused by insufficient storage space.

[0277] A logic controller dedicated to dictionary compression is designed based on digital circuits, such as a cache including a data array, a logic controller, a dictionary unit selector, and a data tag array, and each set of the data array stores a dictionary. However, the complexity of the design and implementation of this method may lead to a longer development cycle, higher development costs, and more complex verification and testing processes. The embodiment of the present application can conveniently implement a data processing process with a high compression rate and a high compression speed based on the above method flow, which can save development costs and reduce implementation difficulty.

[0278] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0279] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0280] In an exemplary embodiment, as shown in FIG15 , a data processing device is provided, including:

[0281] A first determining module 1501 is used to determine data to be compressed;

[0282] A second determining module 1502 is used to determine a target dictionary corresponding to the data to be compressed according to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0283] The compression module 1503 is used to compress the data to be compressed using the target dictionary to obtain compressed data.

[0284] In one embodiment, the second determining module 1502 includes:

[0285] An acquiring unit, configured to acquire a first Key corresponding to the data to be compressed according to the data to be compressed;

[0286] A determining unit is configured to, if a target Value corresponding to the first Key is determined in a first key-value pair according to the first Key, use a dictionary corresponding to the target Value as the target dictionary.

[0287] In one of the embodiments, the acquisition unit is specifically used to obtain a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed; and use the first identifier, the first offset position, and the first number of data blocks as the first Key.

[0288] In one of the embodiments, the target Value includes a dictionary identifier of the target dictionary.

[0289] In one of the embodiments, before determining the data to be compressed, the device further includes:

[0290] A first writing module, used for receiving a data writing request, and storing the writing data corresponding to the data writing request in a disk;

[0291] The first determination module 1501 is specifically configured to determine the data to be compressed from the write data stored in the disk.

[0292] In one embodiment, the first determination module 1501 is specifically used to determine the write data stored in the first storage location in the disk as the data to be compressed according to a compression strategy, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a sequential order of write times of each storage location, and a compression flag of the write data.

[0293] In one embodiment, the compression module 1503 is specifically used to load the data to be compressed from the disk to the memory, and load the target dictionary from the disk to the memory; and compress the data to be compressed using the target dictionary in the memory to obtain the compressed data.

[0294] In one embodiment, the device further comprises:

[0295] A third determination module is used to determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to the target training data, wherein each target training data has the same number of data blocks as the data to be compressed;

[0296] A mapping module is used to generate a Key for each of the dictionaries according to the target training data, determine the Value corresponding to the dictionary, and use a preset key-value storage library to establish a key-value pair between the Key and the Value, each of the key-value pairs including the first key-value pair.

[0297] In one embodiment, the third determination module is specifically used to detect whether there is historical training data whose similarity with the target training data is greater than a preset threshold; if the historical training data exists, the dictionary corresponding to the historical training data is determined as the dictionary corresponding to the target training data; if the historical training data does not exist, the dictionary is obtained by training according to the target training data.

[0298] In one embodiment, the third determination module is specifically used to preprocess the target training data to obtain preprocessed data; perform frequency statistics on the preprocessed data to obtain frequency statistical results; construct a high-frequency vocabulary dictionary based on the frequency statistical results, and use the high-frequency vocabulary dictionary as the dictionary corresponding to the target training data.

[0299] In one embodiment, the device further comprises:

[0300] The second writing module is used to write each of the key-value pairs into the write-ahead log.

[0301] In one embodiment, the device further comprises:

[0302] The storage module is used to store the compressed data in a second storage location in the disk, and to establish a data access path corresponding to the compressed data according to the second storage location to replace the data access path corresponding to the data to be compressed.

[0303] In one embodiment, the device further comprises:

[0304] A replacement module is used to generate a second Key according to the compressed data, and use the second Key to replace the first Key in the first key-value pair to obtain a second key-value pair.

[0305] In one embodiment, the device further comprises:

[0306] The first recycling module is used to mark the data to be compressed as garbage data and perform garbage recycling on the garbage data during the garbage recycling process.

[0307] In one embodiment, the device further comprises:

[0308] An acquisition module, configured to acquire, according to the compressed data, the second Key corresponding to the compressed data when the compressed data needs to be decompressed;

[0309] a fourth determining module, configured to determine, according to the second Key, the target Value corresponding to the second Key in the second key-value pair;

[0310] A decompression module is used to decompress the compressed data using the target dictionary corresponding to the target Value.

[0311] In one embodiment, the device further comprises:

[0312] An accumulation module, used for accumulating 1 to the usage frequency count value of the target dictionary;

[0313] The second recycling module is used to delete the target dictionary and the second key-value pair if the usage frequency count value is less than a preset threshold within a preset time period.

[0314] In an exemplary embodiment, as shown in FIG16 , a data processing device is provided, including:

[0315] A first determining module 1601 is used to determine compressed data;

[0316] A second determining module 1602 is used to determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the to-be-compressed data corresponding to the compressed data;

[0317] The decompression module 1603 is used to decompress the compressed data using the target dictionary.

[0318] In one embodiment, the second determining module 1602 includes:

[0319] An acquiring unit, configured to acquire, according to the compressed data, a second Key corresponding to the compressed data;

[0320] A determination unit is used to determine a target Value corresponding to the second Key in a second key-value pair according to the second Key, and use a dictionary corresponding to the target Value as the target dictionary.

[0321] In one embodiment, the acquisition unit is specifically used to obtain a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data; and use the second identifier, the second offset position, and the second number of data blocks as the second Key.

[0322] In one of the embodiments, the target Value includes a dictionary identifier of the target dictionary.

[0323] In one embodiment, the device further comprises:

[0324] An accumulation module, used for accumulating 1 to the usage frequency count value of the target dictionary;

[0325] A recycling module is used to delete the target dictionary and the second key-value pair if the usage frequency count value is less than a preset threshold within a preset time period.

[0326] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0327] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG17. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0328] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG18. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0329] Those skilled in the art will understand that the structures shown in FIG. 17 and FIG. 18 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0330] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0331] Determine the data to be compressed;

[0332] Determine, according to the data to be compressed, a target dictionary corresponding to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0333] The target dictionary is used to compress the data to be compressed to obtain compressed data.

[0334] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0335] According to the data to be compressed, obtaining a first Key corresponding to the data to be compressed;

[0336] If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

[0337] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0338] Acquire a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed;

[0339] The first identifier, the first offset position, and the first data block quantity are used as the first Key.

[0340] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0341] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0342] Receive a data write request, and store the write data corresponding to the data write request in a disk;

[0343] The step of determining the data to be compressed includes:

[0344] The data to be compressed is determined from the written data stored in the disk.

[0345] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0346] According to the compression strategy, the written data stored in the first storage location in the disk is determined as the data to be compressed, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a writing time sequence of each storage location, and a compression mark of the written data.

[0347] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0348] Loading the data to be compressed from the disk into the memory, and loading the target dictionary from the disk into the memory;

[0349] The target dictionary is used to compress the data to be compressed in the memory to obtain the compressed data.

[0350] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0351] Determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to each target training data, wherein each target training data has the same number of data blocks as the data to be compressed;

[0352] For each of the dictionaries, a Key is generated according to the target training data, a Value corresponding to the dictionary is determined, and a key-value pair between the Key and the Value is established using a preset key-value storage library, each of the key-value pairs including the first key-value pair.

[0353] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0354] Detecting whether there is historical training data whose similarity to the target training data is greater than a preset threshold;

[0355] If the historical training data exists, determining the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data;

[0356] If the historical training data does not exist, the dictionary is obtained by training according to the target training data.

[0357] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0358] Preprocessing the target training data to obtain preprocessed data;

[0359] Performing frequency statistics on the preprocessed data to obtain frequency statistics results;

[0360] A high-frequency vocabulary dictionary is constructed according to the frequency statistics result, and the high-frequency vocabulary dictionary is used as the dictionary corresponding to the target training data.

[0361] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0362] Each of the key-value pairs is written into a write-ahead log.

[0363] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0364] The compressed data is stored in a second storage location in the disk, and a data access path corresponding to the compressed data is established according to the second storage location to replace the data access path corresponding to the data to be compressed.

[0365] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0366] A second Key is generated according to the compressed data, and the second Key is used to replace the first Key in the first key-value pair to obtain a second key-value pair.

[0367] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0368] The data to be compressed is marked as garbage data, and during a garbage collection process, garbage collection is performed on the garbage data.

[0369] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0370] When the compressed data needs to be decompressed, obtaining the second Key corresponding to the compressed data according to the compressed data;

[0371] According to the second Key, determine the target Value corresponding to the second Key in the second key-value pair;

[0372] The compressed data is decompressed using the target dictionary corresponding to the target Value.

[0373] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0374] The usage frequency count value of the target dictionary is accumulated by 1;

[0375] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0376] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0377] Determine the compressed data;

[0378] Determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data;

[0379] The compressed data is decompressed using the target dictionary.

[0380] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0381] According to the compressed data, obtaining a second Key corresponding to the compressed data;

[0382] According to the second Key, a target Value corresponding to the second Key is determined in a second key-value pair, and a dictionary corresponding to the target Value is used as the target dictionary.

[0383] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented:

[0384] Acquire a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data;

[0385] The second identifier, the second offset position, and the second data block quantity are used as the second Key.

[0386] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0387] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0388] The usage frequency count value of the target dictionary is accumulated by 1;

[0389] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0390] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0391] Determine the data to be compressed;

[0392] Determine, according to the data to be compressed, a target dictionary corresponding to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0393] The target dictionary is used to compress the data to be compressed to obtain compressed data.

[0394] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0395] According to the data to be compressed, obtaining a first Key corresponding to the data to be compressed;

[0396] If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

[0397] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0398] Acquire a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed;

[0399] The first identifier, the first offset position, and the first data block quantity are used as the first Key.

[0400] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0401] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0402] Receive a data write request, and store the write data corresponding to the data write request in a disk;

[0403] The step of determining the data to be compressed includes:

[0404] The data to be compressed is determined from the written data stored in the disk.

[0405] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0406] According to the compression strategy, the written data stored in the first storage location in the disk is determined as the data to be compressed, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a writing time sequence of each storage location, and a compression mark of the written data.

[0407] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0408] Loading the data to be compressed from the disk into the memory, and loading the target dictionary from the disk into the memory;

[0409] The target dictionary is used to compress the data to be compressed in the memory to obtain the compressed data.

[0410] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0411] Determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to each target training data, wherein each target training data has the same number of data blocks as the data to be compressed;

[0412] For each of the dictionaries, a Key is generated according to the target training data, a Value corresponding to the dictionary is determined, and a key-value pair between the Key and the Value is established using a preset key-value storage library, each of the key-value pairs including the first key-value pair.

[0413] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0414] Detecting whether there is historical training data whose similarity to the target training data is greater than a preset threshold;

[0415] If the historical training data exists, determining the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data;

[0416] If the historical training data does not exist, the dictionary is obtained by training according to the target training data.

[0417] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0418] Preprocessing the target training data to obtain preprocessed data;

[0419] Performing frequency statistics on the preprocessed data to obtain frequency statistics results;

[0420] A high-frequency vocabulary dictionary is constructed according to the frequency statistics result, and the high-frequency vocabulary dictionary is used as the dictionary corresponding to the target training data.

[0421] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0422] Each of the key-value pairs is written into a write-ahead log.

[0423] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0424] The compressed data is stored in a second storage location in the disk, and a data access path corresponding to the compressed data is established according to the second storage location to replace the data access path corresponding to the data to be compressed.

[0425] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0426] A second Key is generated according to the compressed data, and the second Key is used to replace the first Key in the first key-value pair to obtain a second key-value pair.

[0427] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0428] The data to be compressed is marked as garbage data, and during a garbage collection process, garbage collection is performed on the garbage data.

[0429] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0430] When the compressed data needs to be decompressed, obtaining the second Key corresponding to the compressed data according to the compressed data;

[0431] According to the second Key, determine the target Value corresponding to the second Key in the second key-value pair;

[0432] The compressed data is decompressed using the target dictionary corresponding to the target Value.

[0433] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0434] The usage frequency count value of the target dictionary is accumulated by 1;

[0435] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0436] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0437] Determine the compressed data;

[0438] Determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data;

[0439] The compressed data is decompressed using the target dictionary.

[0440] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0441] According to the compressed data, obtaining a second Key corresponding to the compressed data;

[0442] According to the second Key, a target Value corresponding to the second Key is determined in a second key-value pair, and a dictionary corresponding to the target Value is used as the target dictionary.

[0443] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0444] Acquire a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data;

[0445] The second identifier, the second offset position, and the second data block quantity are used as the second Key.

[0446] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0447] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0448] The usage frequency count value of the target dictionary is accumulated by 1;

[0449] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0450] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0451] Determine the data to be compressed;

[0452] Determine, according to the data to be compressed, a target dictionary corresponding to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed;

[0453] The target dictionary is used to compress the data to be compressed to obtain compressed data.

[0454] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0455] According to the data to be compressed, obtaining a first Key corresponding to the data to be compressed;

[0456] If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

[0457] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0458] Acquire a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed;

[0459] The first identifier, the first offset position, and the first data block quantity are used as the first Key.

[0460] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0461] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0462] Receive a data write request, and store the write data corresponding to the data write request in a disk;

[0463] The step of determining the data to be compressed includes:

[0464] The data to be compressed is determined from the written data stored in the disk.

[0465] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0466] According to the compression strategy, the written data stored in the first storage location in the disk is determined as the data to be compressed, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a writing time sequence of each storage location, and a compression mark of the written data.

[0467] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0468] Loading the data to be compressed from the disk into the memory, and loading the target dictionary from the disk into the memory;

[0469] The target dictionary is used to compress the data to be compressed in the memory to obtain the compressed data.

[0470] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0471] Determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to each target training data, wherein each target training data has the same number of data blocks as the data to be compressed;

[0472] For each of the dictionaries, a Key is generated according to the target training data, a Value corresponding to the dictionary is determined, and a key-value pair between the Key and the Value is established using a preset key-value storage library, each of the key-value pairs including the first key-value pair.

[0473] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0474] Detecting whether there is historical training data whose similarity to the target training data is greater than a preset threshold;

[0475] If the historical training data exists, determining the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data;

[0476] If the historical training data does not exist, the dictionary is obtained by training according to the target training data.

[0477] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0478] Preprocessing the target training data to obtain preprocessed data;

[0479] Performing frequency statistics on the preprocessed data to obtain frequency statistics results;

[0480] A high-frequency vocabulary dictionary is constructed according to the frequency statistics result, and the high-frequency vocabulary dictionary is used as the dictionary corresponding to the target training data.

[0481] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0482] Each of the key-value pairs is written into a write-ahead log.

[0483] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0484] The compressed data is stored in a second storage location in the disk, and a data access path corresponding to the compressed data is established according to the second storage location to replace the data access path corresponding to the data to be compressed.

[0485] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0486] A second Key is generated according to the compressed data, and the second Key is used to replace the first Key in the first key-value pair to obtain a second key-value pair.

[0487] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0488] The data to be compressed is marked as garbage data, and during a garbage collection process, garbage collection is performed on the garbage data.

[0489] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0490] When the compressed data needs to be decompressed, obtaining the second Key corresponding to the compressed data according to the compressed data;

[0491] According to the second Key, determine the target Value corresponding to the second Key in the second key-value pair;

[0492] The compressed data is decompressed using the target dictionary corresponding to the target Value.

[0493] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0494] The usage frequency count value of the target dictionary is accumulated by 1;

[0495] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0496] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0497] Determine the compressed data;

[0498] Determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data;

[0499] The compressed data is decompressed using the target dictionary.

[0500] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0501] According to the compressed data, obtaining a second Key corresponding to the compressed data;

[0502] According to the second Key, a target Value corresponding to the second Key is determined in a second key-value pair, and a dictionary corresponding to the target Value is used as the target dictionary.

[0503] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0504] Acquire a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data;

[0505] The second identifier, the second offset position, and the second data block quantity are used as the second Key.

[0506] In one embodiment, the target Value includes a dictionary identifier of the target dictionary.

[0507] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0508] The usage frequency count value of the target dictionary is accumulated by 1;

[0509] If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

[0510] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0511] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0512] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0513] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the data to be compressed; Determine, according to the data to be compressed, a target dictionary corresponding to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed; The target dictionary is used to compress the data to be compressed to obtain compressed data.

2. The method according to claim 1, characterized in that The step of determining a target dictionary corresponding to the data to be compressed according to the data to be compressed includes: According to the data to be compressed, obtaining a first Key corresponding to the data to be compressed; If the target Value corresponding to the first Key is determined in the first key-value pair according to the first Key, the dictionary corresponding to the target Value is used as the target dictionary.

3. The method according to claim 2, characterized in that The obtaining, according to the data to be compressed, a first Key corresponding to the data to be compressed includes: Acquire a first identifier of a first logical unit storing the data to be compressed, a first offset position of the data to be compressed in the first logical unit, and a first number of data blocks corresponding to the data to be compressed; The first identifier, the first offset position, and the first data block quantity are used as the first Key.

4. The method according to claim 2, characterized in that: Before determining the data to be compressed, the method further includes: Receive a data write request, and store the write data corresponding to the data write request in a disk; The step of determining the data to be compressed includes: The data to be compressed is determined from the written data stored in the disk.

5. The method according to claim 4, characterized in that The step of determining the data to be compressed from the written data stored in the disk comprises: According to the compression strategy, the written data stored in the first storage location in the disk is determined as the data to be compressed, wherein the compression strategy includes at least one of a preset number of data blocks, a preset data volume, a sequence of writing times of each storage location, and a compression mark of the written data.

6. The method according to claim 4, characterized in that The compressing the data to be compressed by using the target dictionary to obtain compressed data includes: Loading the data to be compressed from the disk into the memory, and loading the target dictionary from the disk into the memory; The target dictionary is used to compress the data to be compressed in the memory to obtain the compressed data.

7. The method according to claim 2, characterized in that Before determining the data to be compressed, the method further includes: Determine a plurality of target training data from the written data stored in the disk, and determine a dictionary corresponding to each target training data according to each target training data, wherein each target training data has the same number of data blocks as the data to be compressed; For each of the dictionaries, a Key is generated according to the target training data, a Value corresponding to the dictionary is determined, and a key-value pair between the Key and the Value is established using a preset key-value storage library, each of the key-value pairs including the first key-value pair.

8. The method according to claim 7, characterized in that The step of determining a dictionary corresponding to each target training data according to each target training data comprises: Detecting whether there is historical training data whose similarity to the target training data is greater than a preset threshold; If the historical training data exists, determining the dictionary corresponding to the historical training data as the dictionary corresponding to the target training data; If the historical training data does not exist, the dictionary is obtained by training according to the target training data.

9. The method according to any one of claims 2 to 8, characterized in that: After the target dictionary is used to compress the data to be compressed to obtain compressed data, the method further includes: A second Key is generated according to the compressed data, and the second Key is used to replace the first Key in the first key-value pair to obtain a second key-value pair.

10. The method according to claim 9, characterized in that The method further comprises: When the compressed data needs to be decompressed, obtaining the second Key corresponding to the compressed data according to the compressed data; According to the second Key, determine the target Value corresponding to the second Key in the second key-value pair; The compressed data is decompressed using the target dictionary corresponding to the target Value.

11. The method according to claim 10, characterized in that After decompressing the compressed data using the target dictionary corresponding to the target Value, the method further includes: The usage frequency count value of the target dictionary is accumulated by 1; If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

12. A data processing method, characterized in that: The method comprises: Determine the compressed data; Determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data; The compressed data is decompressed using the target dictionary.

13. The method according to claim 12, characterized in that The step of determining, according to the compressed data, a target dictionary corresponding to the compressed data includes: According to the compressed data, obtaining a second Key corresponding to the compressed data; According to the second Key, a target Value corresponding to the second Key is determined in a second key-value pair, and a dictionary corresponding to the target Value is used as the target dictionary.

14. The method according to claim 13, characterized in that The obtaining, according to the compressed data, a second Key corresponding to the compressed data, comprises: Acquire a second identifier of a second logical unit storing the compressed data, a second offset position of the compressed data in the second logical unit, and a second number of data blocks corresponding to the compressed data; The second identifier, the second offset position, and the second data block quantity are used as the second Key.

15. The method according to claim 13 or 14, characterized in that After decompressing the compressed data using the target dictionary, the method further includes: The usage frequency count value of the target dictionary is accumulated by 1; If the usage frequency count value is less than a preset threshold value within a preset time period, the target dictionary and the second key-value pair are deleted.

16. A data processing device, characterized in that: The device comprises: A first determining module, used to determine the data to be compressed; A second determination module, configured to determine a target dictionary corresponding to the data to be compressed according to the data to be compressed, wherein the target dictionary is determined in advance according to the data to be compressed; The compression module is used to compress the data to be compressed using the target dictionary to obtain compressed data.

17. A data processing device, characterized in that: The device comprises: A first determining module, used to determine compressed data; A second determination module, configured to determine, according to the compressed data, a target dictionary corresponding to the compressed data, wherein the target dictionary is determined in advance according to the data to be compressed corresponding to the compressed data; A decompression module is used to decompress the compressed data using the target dictionary.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images