Data processing method and device, equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, equipment, a readable storage medium and a program product. The method comprises the steps that data to be compressed are obtained, compression processing can be conducted on the data to be compressed to obtain a compression unit, the compression unit is written into a first target data block in a first storage space, a compression label corresponding to the data to be compressed is generated, and metadata corresponding to the first target data block can be updated according to the compression label, the compression unit comprises compression data corresponding to the to-be-compressed data and compression head information corresponding to the compression data, and the compression label is used for reading the to-be-compressed data. By adopting the method, the data scale of the metadata can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, apparatus, device, readable storage medium and program product. Background Art

[0002] In a storage medium, data is usually stored in allocation units of data blocks. In some cases, in order to improve the utilization rate of the storage medium, the data in each allocation space is compressed to obtain a compression unit corresponding to each allocation space.

[0003] In the related art, when a compression unit with a smaller data volume is written into a data block, compression information related to the compression unit, such as the type of compression algorithm used, etc., will be recorded in the metadata corresponding to the data block.

[0004] However, the above method has the problem of large data size of metadata. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, equipment, readable storage medium and program product that can reduce the data size of metadata in response to the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Get the data to be compressed;

[0008] Compressing the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0009] The compression unit is written into the first target data block in the first storage space, and a compression tag corresponding to the data to be compressed is generated. The metadata corresponding to the first target data block is updated according to the compression tag, and the compression tag is used to read the data to be compressed.

[0010] In one embodiment, compressing the data to be compressed to obtain a compression unit includes:

[0011] Determine the optimal compression algorithm corresponding to the data to be compressed;

[0012] The data to be compressed is compressed according to an optimal compression algorithm to obtain compressed data, and compression header information is generated according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0013] In one embodiment, determining an optimal compression algorithm corresponding to the data to be compressed includes:

[0014] According to a plurality of candidate compression algorithms, the data to be compressed are respectively compressed to obtain a plurality of candidate compressed data;

[0015] The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

[0016] In one embodiment, the method further comprises:

[0017] Writing the compression unit to the second storage space at a first write position in the first target data block;

[0018] Detecting whether the data amount in the first target data block reaches a preset data amount threshold;

[0019] If the amount of data in the first target data block reaches a preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0020] In one embodiment, the method further comprises:

[0021] Obtaining a data read request, where the data read request includes a data identifier of the data to be read;

[0022] Determine a second target data block according to the data identifier, and determine whether the data to be read is compressed data according to a compression tag stored in metadata corresponding to the second target data block;

[0023] If the data to be read is compressed data, the second write position of the target compression unit corresponding to the data to be read is obtained from the compression tail included in the second target data block, and the data to be read is read according to the second write position.

[0024] In one embodiment, reading the to-be-read data according to the second write position includes:

[0025] Reading a target compression unit corresponding to the to-be-read data from the second writing position;

[0026] Decompressing the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data;

[0027] The data to be read is determined according to the decompressed data.

[0028] In a second aspect, the present application further provides a data processing device, comprising:

[0029] An acquisition module, used for acquiring data to be compressed;

[0030] A compression module, used for compressing the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0031] The writing module is used to write the compression unit into the first target data block in the first storage space, and generate a compression tag corresponding to the data to be compressed, and update the metadata corresponding to the first target data block according to the compression tag, and the compression tag is used to read the data to be compressed.

[0032] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect described above.

[0034] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

[0035] The above-mentioned data processing method, device, equipment, readable storage medium and program product can compress the data to be compressed by acquiring the data to be compressed, obtain the compression unit, write the compression unit into the first target data block in the first storage space, and generate a compression tag corresponding to the data to be compressed, and update the metadata corresponding to the first target data block according to the compression tag, wherein the compression unit includes the compressed data corresponding to the data to be compressed and the compression header information corresponding to the compressed data, and the compression tag is used to read the data to be compressed. In this way, the compression header information is set in the compression unit, so that the information related to the compression unit that is originally to be written into the metadata is written into the compression header information included in each compression unit itself, and the metadata only needs to record the compression tag used to read the compressed data, which reduces the data scale of the metadata and further improves the efficiency of metadata update. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the structure of data blocks in traditional technology;

[0038] Figure 2 An application environment diagram of a data processing method in an embodiment;

[0039] Figure 3 is a flow chart of a data processing method in one embodiment;

[0040] Figure 4 is a flow chart of step 302 in another embodiment;

[0041] Figure 5 A schematic flow chart of a storage process of each compression unit writing a position in another embodiment;

[0042] Figure 6 is a schematic diagram of the structure of a first target data block in another embodiment;

[0043] Figure 7 A schematic diagram of a process of reading compressed data in another embodiment;

[0044] Figure 8 is a flowchart of an exemplary data processing method in another embodiment;

[0045] Fig. 9 is a structural block diagram of a data processing device in one embodiment;

[0046] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In storage media, data is usually stored in allocation units of data blocks. The specific structure can be found in Figure 1 , the allocation units in the storage medium belong to a data block in a logical relationship, but are in different storage locations in a physical relationship.

[0049] In some cases, in order to improve the utilization rate of the storage medium, the data in each allocated space is compressed to obtain a compression unit corresponding to each allocated space.

[0050] In the related art, when a compression unit with a smaller data volume is written into a data block, compression information related to the compression unit, such as the type of compression algorithm used, will be recorded in the metadata corresponding to the data block. However, when the compression unit is small, the number of compression units in the data block increases, and the amount of data recorded in the metadata becomes very large, which will lead to a decrease in the efficiency of metadata updates. Increasing the amount of data in the compression unit can indeed reduce the number of compression units, thereby reducing the amount of data in the metadata. However, when the data in the compression unit needs to be read, the amount of data that needs to be decompressed increases, causing serious read amplification.

[0051] In view of this, the present application provides a data processing method, device, equipment, readable storage medium and program product, which can obtain the data to be compressed, compress the data to be compressed, obtain the compression unit, write the compression unit into the first target data block in the first storage space, and generate the compression tag corresponding to the data to be compressed, and update the metadata corresponding to the first target data block according to the compression tag, wherein the compression unit includes the compressed data corresponding to the data to be compressed and the compression header information corresponding to the compressed data, and the compression tag is used to read the data to be compressed. In this way, the compression header information is set in the compression unit, so that the information related to the compression unit that is originally to be written into the metadata is written into the compression header information included in each compression unit itself, and the metadata only needs to record the compression tag used to read the compressed data, which reduces the data scale of the metadata and improves the efficiency of metadata update.

[0052] The data processing method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown. Among them, the data storage system can store the data that the server 201 needs to process. The data storage system can be integrated on the server 201, or it can be placed on the cloud or other network servers. The server 201 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In an exemplary embodiment, Figure 3 As shown, a data processing method is provided, which is applied to Figure 2 The server 201 in the example is used as an example to illustrate, including the following steps 301 to 303. Among them:

[0054] Step 301: Obtain data to be compressed.

[0055] The data to be compressed may be data waiting to be compressed. In a possible implementation, the data to be compressed may be data already stored in a storage medium. The server may obtain the data to be compressed according to the received write position.

[0056] In another possible implementation, the data to be compressed may be obtained by the server through an external input device. In an embodiment of the present application, the server may obtain a data write request through an external input device, and the data write request includes the data to be written; after receiving the data write request, the server may write the data to be written in the data write request into a log, and write the data to be written in the log into a storage space waiting to be written, which may be in a memory or a cache; when the amount of data to be written in the storage space waiting to be written reaches a preset data amount threshold, such as the amount of data of an allocation unit or an integer multiple of the amount of data of an allocation unit, the server may start writing the data to be written into the storage medium.

[0057] Regarding the process of writing to the storage medium, in an embodiment of the present application, the server can submit each data to be written as a write task to the task queue of the background, and the background will continuously perform dequeue operations on the task queue to obtain write tasks; for the data to be written included in a write task, the server can detect whether the data to be written is data that needs to be compressed. If the detection result is that compression is required, the server can determine that the data to be written is data to be compressed, otherwise, the server determines that the data to be written is not data to be compressed.

[0058] Regarding the process in which the server detects whether the data to be written is data that needs to be compressed, optionally, the write task carries an indication mark indicating whether the data to be written in the current write task is compressed, and the server determines whether the data to be written is data to be compressed based on the indication mark; optionally, the server can compress the data to be written according to a preset compression algorithm, and detect whether the amount of data to be written after compression is less than the amount of data before compression. If so, the server can determine that the data to be written is data to be compressed.

[0059] Step 302: compress the data to be compressed to obtain a compression unit.

[0060] The compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data.

[0061] The compressed data may be data obtained after compression processing is performed on the data to be compressed, and the compressed header information may include compression information related to the compressed data. In an embodiment of the present application, additional compressed header information is added to the compression unit, and the compressed information originally written in the metadata is written into the compressed header information included in the compression unit, thereby reducing the data size in the metadata.

[0062] Regarding the process of the server compressing the data to be compressed, optionally, the server compresses the data to be compressed according to a preset compression algorithm. Optionally, the server also obtains the corresponding compression algorithm designation identifier when obtaining the data to be compressed. The server can determine the target compression algorithm from multiple candidate compression algorithms according to the compression algorithm designation identifier, and use the target compression algorithm to compress the data to be compressed; optionally, the server can determine the optimal compression algorithm from multiple candidate compression algorithms according to preset conditions, and use the optimal compression algorithm to compress the data to be compressed.

[0063] Step 303: write the compression unit into the first target data block in the first storage space, generate a compression tag corresponding to the data to be compressed, and update metadata corresponding to the first target data block according to the compression tag.

[0064] The compression tag is used to read the data to be compressed.

[0065] The first target data block is a data block waiting for data to be written, and the first storage space may be in a storage medium, which may be a magnetic disk or a solid state drive.

[0066] The server can write the compression unit into the first target data block and update the metadata corresponding to the first target data block. In an embodiment of the present application, the server can generate a unit identifier corresponding to the current compression unit, and the unit identifier can uniquely identify the compressed data in the compression unit. The server can associate the unit identifier with its corresponding compression tag and write it into the metadata. The compression tag can be just a 1kb mark, which can be used to indicate that the data in the compression unit is compressed data.

[0067] In the above embodiment, compression header information is set in the compression unit, so that the information related to the compression unit that is originally to be written into the metadata is written into the compression header information included in each compression unit itself. The metadata only needs to record the compression tag used to read the compressed data, which reduces the data size of the metadata and improves the efficiency of metadata updating.

[0068] In one embodiment, based on the above Figure 3 The embodiment shown, see Figure 4 This embodiment relates to a process of compressing data to be compressed to obtain a compression unit. Figure 4 As shown, step 302 may include step 401 and step 402 .

[0069] Step 401, determining the optimal compression algorithm corresponding to the data to be compressed.

[0070] In an embodiment of the present application, the optimal compression algorithm may be a compression algorithm that is most optimized for compressing the data to be compressed. Regarding the determination of the optimal compression algorithm, in a possible implementation manner, when the server obtains the data to be compressed, it will also obtain its corresponding optimal compression algorithm designation identifier. The server can determine the optimal compression algorithm from multiple candidate compression algorithms based on the optimal compression algorithm designation identifier, and use the target compression algorithm to compress the data to be compressed.

[0071] In another possible implementation, the server may compress the data to be compressed according to multiple candidate compression algorithms to obtain multiple candidate compressed data, and then determine the candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data as the target compression algorithm.

[0072] In an embodiment of the present application, the candidate compression algorithms may be Huffman algorithm, run length coding, arithmetic coding, and the like.

[0073] Step 402: compress the data to be compressed according to the optimal compression algorithm to obtain compressed data, and generate compression header information according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0074] The server compresses the data to be compressed using the optimal compression algorithm and obtains the compressed data. Then, the server can use the algorithm name of the optimal compression algorithm or the algorithm identifier that uniquely identifies the algorithm, the data length of the data to be compressed, and the data length of the compressed data as the current compression but with corresponding compression header information.

[0075] In one embodiment, based on the above Figure 3 The embodiment shown, see Figure 5 , this embodiment involves the storage process of each compression unit writing position. Figure 5 As shown, the process may include steps 501 to 503.

[0076] Step 501: write a compression unit into a second storage space at a first write position in a first target data block.

[0077] Step 502: Detect whether the data volume in the first target data block reaches a preset data volume threshold.

[0078] Step 503: if the amount of data in the first target data block reaches a preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0079] In a possible implementation, when the server writes the compression unit into the first target data block, it also obtains its first write position in the first target data block. In an embodiment of the present application, the first write position may be the position of the compression header information of the compression unit in the first target data block.

[0080] After acquiring the first write position, the server may write the first write position into a second storage space. The storage space where the second storage space is located is different from the first storage space. The second storage space may be a cache or a memory.

[0081] After writing the first write position into the second storage space, the server may detect whether the amount of data in the first target data block reaches a preset data amount threshold, where the preset data amount threshold may be the amount of data that the first target data block can accommodate.

[0082] Optionally, if the amount of data in the first target data block does not reach a preset data amount threshold, the server can continue to execute the process of writing other data into the first target data block; optionally, if the amount of data in the first target data block reaches a preset data amount threshold, the server can obtain the write position stored in the second storage space, and write the write position into the compression tail included in the first target data block.

[0083] In the embodiment of the present application, a compression tail is added to the modulus of the first target data block, and the compression tail records the write position corresponding to each compression unit, which facilitates the subsequent reading of data in the compression unit. Based on this embodiment, the structure of the first target data block can refer to Figure 6 .

[0084] In one embodiment, based on the above Figure 5 The embodiment shown, see Figure 7 , this embodiment involves the process of reading compressed data. Figure 7 As shown, the process may include steps 701 to 703.

[0085] Step 701: Obtain a data read request.

[0086] Optionally, the server may obtain the data reading request through an external input device. Optionally, the server may obtain the data reading request generated within the system due to certain specific data reading tasks.

[0087] In an embodiment of the present application, a data read request may include a data identifier of the data to be read, which may uniquely identify the data to be read. Optionally, the data identifier may be a hash value corresponding to the data to be read. Optionally, the data identifier may be a preset field.

[0088] In one possible implementation, before step 702, the server may detect whether the data to be read corresponding to the data identifier exists in the cache. For example, the server may first try to read the data to be read from the cache. If the data to be read is successfully read, it means that the data to be read exists in the second storage space, and the server may respond to the data read request based on the read data. If the data to be read is not successfully read, the server may determine that the data to be read exists in the first storage space, and the server may execute step 702.

[0089] Step 702: determine the second target data block according to the data identifier, and determine whether the data to be read is compressed data according to the compression tag stored in the metadata corresponding to the second target data block.

[0090] After obtaining the data identifier in the data reading request, the server can optionally detect the metadata corresponding to each candidate data block based on the data identifier, so as to determine the second target data block to which the data to be read belongs; optionally, the server can determine the target mapping relationship corresponding to the data identifier from multiple candidate mapping relationships based on the data identifier, and determine the second target data block based on the target mapping relationship.

[0091] After determining the second target data block, the server may read the compression tag corresponding to the data to be read from the second target data block according to the data tag, and determine whether the data to be read is compressed data according to the compression tag.

[0092] In a possible implementation, the server determines that the data to be read is not compressed data according to the compression tag, and the server may directly read the data to be read from the second target data block, and respond to the data read request according to the read data to be read.

[0093] Step 703: If the data to be read is compressed data, obtain the second write position of the target compression unit corresponding to the data to be read from the compression tail included in the second target data block, and read the data to be read according to the second write position.

[0094] In one possible implementation, if the server determines that the data to be read is compressed data, the server can obtain the second write position of the target compression unit to which the data to be read belongs in the second target data block from the compression tail included in the second target data block according to the data identifier. Based on the second write position, the server can read the target compression unit and obtain the data to be read based on the target compression unit.

[0095] Based on the above embodiment, reading the data to be read according to the second write position may include: the server can read the target compression unit corresponding to the data to be read from the second write position, and decompress the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain the decompressed data, and then determine the data to be read based on the decompressed data.

[0096] In an embodiment of the present application, the target compression header information included in the target compression unit includes relevant compression information, such as compression algorithm, data length before compression, data length after compression, etc. The server can decompress the target compressed data included in the target compression unit according to the compression algorithm to obtain the decompressed data. Optionally, the decompressed data can also be security checked according to the data length before compression and the data length after compression.

[0097] In one embodiment, referring to Figure 8 , provides an exemplary data processing method, which can be applied to Figure 1 In the implementation environment shown.

[0098] Step 801, obtaining data to be compressed.

[0099] Step 802: compress the data to be compressed according to multiple candidate compression algorithms to obtain multiple candidate compressed data.

[0100] Step 803: determine the candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data as the target compression algorithm.

[0101] Step 804: compress the data to be compressed according to the optimal compression algorithm to obtain compressed data, and generate compression header information according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0102] The compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data.

[0103] Step 805: write the compression unit into the first target data block in the first storage space, generate a compression tag corresponding to the data to be compressed, and update metadata corresponding to the first target data block according to the compression tag.

[0104] The compression tag is used to read the data to be compressed.

[0105] Step 806: Write the first write position of the compression unit in the first target data block into the second storage space.

[0106] Step 807: Detect whether the data volume in the first target data block reaches a preset data volume threshold.

[0107] Step 808: If the amount of data in the first target data block reaches a preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0108] Step 809: Obtain a data read request, where the data read request includes a data identifier of the data to be read.

[0109] Step 810: determine the second target data block according to the data identifier, and determine whether the data to be read is compressed data according to the compression tag stored in the metadata corresponding to the second target data block.

[0110] Step 811: If the data to be read is compressed data, obtain the second write position of the target compression unit corresponding to the data to be read from the compression tail included in the second target data block.

[0111] Step 812: read the target compression unit corresponding to the to-be-read data from the second write position.

[0112] Step 813: decompress the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data.

[0113] Step 814, determining the data to be read according to the decompressed data.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0116] In an exemplary embodiment, Fig. 9As shown, a data processing device is provided, including: an acquisition module 901, a compression module 902 and a writing module 903, wherein:

[0117] An acquisition module 901 is used to acquire data to be compressed;

[0118] A compression module 902 is used to compress the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0119] The writing module 903 is used to write the compression unit into the first target data block in the first storage space, and generate a compression tag corresponding to the data to be compressed, and update the metadata corresponding to the first target data block according to the compression tag, and the compression tag is used to read the data to be compressed.

[0120] In one embodiment, the compression module 902 includes:

[0121] An algorithm determination unit, used to determine the optimal compression algorithm corresponding to the data to be compressed;

[0122] A compression processing unit is used to compress the data to be compressed according to the optimal compression algorithm to obtain the compressed data, and generate the compression header information according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0123] In one embodiment, the algorithm determination unit is specifically configured to perform:

[0124] Compressing the data to be compressed according to multiple candidate compression algorithms respectively to obtain multiple candidate compressed data;

[0125] The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

[0126] In one embodiment, the apparatus further comprises:

[0127] a position cache module, configured to write the first write position of the compression unit in the first target data block into a second storage space;

[0128] A threshold detection module, used to detect whether the data amount in the first target data block reaches a preset data amount threshold;

[0129] The compression tail writing module is used to write each writing position stored in the second storage space into the compression tail included in the first target data block if the data amount in the first target data block reaches the preset data amount threshold.

[0130] In one embodiment, the apparatus further comprises:

[0131] A request acquisition module, used to acquire a data reading request, wherein the data reading request includes a data identifier of the data to be read;

[0132] A compression judgment module, used to determine the second target data block according to the data identifier, and determine whether the data to be read is compressed data according to the compression tag stored in the metadata corresponding to the second target data block;

[0133] A reading module is used to obtain a second write position of a target compression unit corresponding to the data to be read from a compression tail included in the second target data block if the data to be read is compressed data, and read the data to be read according to the second write position.

[0134] In one embodiment, the reading module comprises:

[0135] A compression unit reading unit, configured to read the target compression unit corresponding to the to-be-read data from the second writing position;

[0136] a decompression unit, configured to decompress the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data;

[0137] A data determination unit is used to determine the data to be read according to the decompressed data.

[0138] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0140] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0142] Get the data to be compressed;

[0143] Performing compression processing on the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0144] The compression unit is written into a first target data block in a first storage space, and a compression tag corresponding to the data to be compressed is generated. The metadata corresponding to the first target data block is updated according to the compression tag, and the compression tag is used to read the data to be compressed.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] Determining an optimal compression algorithm corresponding to the data to be compressed;

[0147] The data to be compressed is compressed according to the optimal compression algorithm to obtain the compressed data, and the compression header information is generated according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] Compressing the data to be compressed according to multiple candidate compression algorithms respectively to obtain multiple candidate compressed data;

[0150] The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] Writing the first write position of the compression unit in the first target data block into the second storage space;

[0153] Detecting whether the amount of data in the first target data block reaches a preset data amount threshold;

[0154] If the amount of data in the first target data block reaches the preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] Obtaining a data read request, wherein the data read request includes a data identifier of data to be read;

[0157] Determine a second target data block according to the data identifier, and determine whether the data to be read is compressed data according to a compression tag stored in metadata corresponding to the second target data block;

[0158] If the data to be read is compressed data, the second write position of the target compression unit corresponding to the data to be read is obtained from the compression tail included in the second target data block, and the data to be read is read according to the second write position.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] Reading the target compression unit corresponding to the to-be-read data from the second writing position;

[0161] Decompressing the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data;

[0162] The data to be read is determined according to the decompressed data.

[0163] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0164] Get the data to be compressed;

[0165] Performing compression processing on the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0166] The compression unit is written into a first target data block in a first storage space, and a compression tag corresponding to the data to be compressed is generated. The metadata corresponding to the first target data block is updated according to the compression tag, and the compression tag is used to read the data to be compressed.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0168] Determining an optimal compression algorithm corresponding to the data to be compressed;

[0169] The data to be compressed is compressed according to the optimal compression algorithm to obtain the compressed data, and the compression header information is generated according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0171] Compressing the data to be compressed according to multiple candidate compression algorithms respectively to obtain multiple candidate compressed data;

[0172] The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] Writing the first write position of the compression unit in the first target data block into the second storage space;

[0175] Detecting whether the amount of data in the first target data block reaches a preset data amount threshold;

[0176] If the amount of data in the first target data block reaches the preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] Obtaining a data read request, wherein the data read request includes a data identifier of data to be read;

[0179] Determine a second target data block according to the data identifier, and determine whether the data to be read is compressed data according to a compression tag stored in metadata corresponding to the second target data block;

[0180] If the data to be read is compressed data, the second write position of the target compression unit corresponding to the data to be read is obtained from the compression tail included in the second target data block, and the data to be read is read according to the second write position.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Reading the target compression unit corresponding to the to-be-read data from the second writing position;

[0183] Decompressing the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data;

[0184] The data to be read is determined according to the decompressed data.

[0185] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0186] Get the data to be compressed;

[0187] Performing compression processing on the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data;

[0188] The compression unit is written into a first target data block in a first storage space, and a compression tag corresponding to the data to be compressed is generated. The metadata corresponding to the first target data block is updated according to the compression tag, and the compression tag is used to read the data to be compressed.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0190] Determining an optimal compression algorithm corresponding to the data to be compressed;

[0191] The data to be compressed is compressed according to the optimal compression algorithm to obtain the compressed data, and the compression header information is generated according to the optimal compression algorithm, the data to be compressed and the compressed data.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] Compressing the data to be compressed according to multiple candidate compression algorithms respectively to obtain multiple candidate compressed data;

[0194] The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] Writing the first write position of the compression unit in the first target data block into the second storage space;

[0197] Detecting whether the amount of data in the first target data block reaches a preset data amount threshold;

[0198] If the amount of data in the first target data block reaches the preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Obtaining a data read request, wherein the data read request includes a data identifier of data to be read;

[0201] Determine a second target data block according to the data identifier, and determine whether the data to be read is compressed data according to a compression tag stored in metadata corresponding to the second target data block;

[0202] If the data to be read is compressed data, the second write position of the target compression unit corresponding to the data to be read is obtained from the compression tail included in the second target data block, and the data to be read is read according to the second write position.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0204] Reading the target compression unit corresponding to the to-be-read data from the second writing position;

[0205] Decompressing the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data;

[0206] The data to be read is determined according to the decompressed data.

[0207] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0208] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0209] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0210] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Get the data to be compressed; Performing compression processing on the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data; The compression unit is written into a first target data block in a first storage space, and a compression tag corresponding to the data to be compressed is generated. The metadata corresponding to the first target data block is updated according to the compression tag, and the compression tag is used to read the data to be compressed.

2. The method according to claim 1, characterized in that The step of compressing the data to be compressed to obtain a compression unit comprises: Determining an optimal compression algorithm corresponding to the data to be compressed; The data to be compressed is compressed according to the optimal compression algorithm to obtain the compressed data, and the compression header information is generated according to the optimal compression algorithm, the data to be compressed and the compressed data.

3. The method according to claim 2, characterized in that The determining the optimal compression algorithm corresponding to the data to be compressed includes: Compressing the data to be compressed according to multiple candidate compression algorithms respectively to obtain multiple candidate compressed data; The candidate compression algorithm corresponding to the candidate compressed data with the smallest data length among the candidate compressed data is determined as the target compression algorithm.

4. The method according to claim 1, characterized in that The method further comprises: Writing the first write position of the compression unit in the first target data block into the second storage space; Detecting whether the amount of data in the first target data block reaches a preset data amount threshold; If the amount of data in the first target data block reaches the preset data amount threshold, each write position stored in the second storage space is written into the compression tail included in the first target data block.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining a data read request, wherein the data read request includes a data identifier of data to be read; Determine a second target data block according to the data identifier, and determine whether the data to be read is compressed data according to a compression tag stored in metadata corresponding to the second target data block; If the data to be read is compressed data, the second write position of the target compression unit corresponding to the data to be read is obtained from the compression tail included in the second target data block, and the data to be read is read according to the second write position.

6. The method according to claim 5, characterized in that The step of reading the to-be-read data according to the second writing position comprises: Reading the target compression unit corresponding to the to-be-read data from the second writing position; Decompressing the target compressed data included in the target compression unit according to the target compression header information included in the target compression unit to obtain decompressed data; The data to be read is determined according to the decompressed data.

7. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring data to be compressed; A compression module, used for compressing the data to be compressed to obtain a compression unit, wherein the compression unit includes compressed data corresponding to the data to be compressed and compression header information corresponding to the compressed data; A writing module is used to write the compression unit into a first target data block in a first storage space, and generate a compression tag corresponding to the data to be compressed, and update metadata corresponding to the first target data block according to the compression tag, and the compression tag is used to read the data to be compressed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images