Data processing method, device, equipment and computer readable storage medium

Abstract

The present application discloses a data processing method, device, equipment and computer-readable storage medium, and belongs to the field of cloud computing technology. The present application obtains a data processing instruction indicating decompressing target data from first compressed data, the first compressed data is obtained from the data group to which the compressed target data belongs, and the data processing instruction includes the first starting position of the target data in the data group and the length of the target data; based on the first starting position and the reference distance, the second starting position of the second compressed data used for decompressing the target data in the first compressed data is determined, and the reference distance indicates the maximum distance between any data segment in the target data and the data segment it references; the second compressed data is decompressed according to the second starting position and the end position of the second compressed data to obtain the target data, and the end position is determined by the first starting position and the length of the target data. The present application can provide a clear decompression boundary before decompression to improve the decompression accuracy.

Description

[0001] This application claims priority to Chinese patent application No. 202311559827.1, filed on November 20, 2023, and entitled “Data processing method, device, equipment and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0003] With the development of cloud computing technology, the computing power of cloud computing has gradually improved. The amount of data required in the computing process is increasing, and the amount of data that needs to be stored is also increasing. Therefore, in the process of data storage, the data can be compressed and stored to reduce the storage cost of the data. When there is a computing demand for the stored data, the compressed data needs to be decompressed first to obtain the data before compression, and the computing demand can be met by computing the data before compression. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device and computer-readable storage medium, which can provide a clear decompression boundary and achieve accurate decompression. The technical solution is as follows:

[0005] In a first aspect, a data processing method is provided, the method comprising: obtaining a data processing instruction, the data processing instruction instructing to decompress target data from first compressed data, the first compressed data being obtained by compressing a data group to which the target data belongs, the data processing instruction comprising a first starting position of the target data in the data group and a length of the target data; determining a second starting position of second compressed data in the first compressed data based on the first starting position and a reference distance, the second compressed data being used to decompress and obtain the target data, the reference distance indicating a maximum value of a distance between any data segment in the target data and a data segment referenced by any data segment; decompressing the second compressed data according to the second starting position and an end position of the second compressed data to obtain the target data, the end position of the second compressed data being determined based on the first starting position and the length of the target data.

[0006] Since the reference distance indicates the maximum value of the distance between any data segment in the target data and the data segment referenced by any data segment, the data segment referenced by each data segment in the target data may be located in the target data, or in the partial data whose distance from the first starting position is less than or equal to the reference distance. Then, according to the first starting position and the reference distance, the second starting position of the second compressed data used for decompressing the target data can be determined, and it is ensured that the target data can be obtained by decompressing the second compressed data from the second starting position. In addition, according to the first starting position and the length of the target data, the end position of the second compressed data can be accurately determined, and the decompression is stopped at the end position of the second compressed data. Decompression is performed using the determined second starting position and the end position of the second compressed data as the decompression boundary, which not only ensures that the decompressed data includes the target data, but also reduces the decompression of compressed data that is not necessary to be decompressed, improves decompression efficiency and accuracy, and reduces resource waste.

[0007] In a possible implementation, the first compressed data includes multiple compressed segments, the data group includes multiple data segments, and each compressed segment is compressed by the corresponding data segment; based on the first starting position and the reference distance, the second starting position of the second compressed data in the first compressed data is determined, including: obtaining the starting position of each data segment in the data group; according to the starting position of each data segment, determining the first data segment to which the first position belongs among the multiple data segments, the first position is determined based on the difference between the first starting position and the reference distance; according to the starting position of the first data segment, determining the starting position of the first compressed segment corresponding to the first data segment; and determining the starting position of the first compressed segment as the second starting position.

[0008] Due to the difference between the first starting position and the reference distance, it can indicate the first position that is before the first starting position and whose distance from the first starting position is the reference distance, and since the distance between any data segment in the target data and the data segment referenced by any data segment is less than or equal to the reference distance, that is, the data segment referenced by any data segment in the target data segment is located at the first position or after the first position, and thus decompression is started from the starting position of the first compressed segment corresponding to the first data segment belonging to the first position, so that the target data can be obtained by decompression.

[0009] In one possible implementation, the starting position of each data segment in the data group is obtained, including: obtaining the length of each data segment; determining the starting position of each data segment in the data group according to the length of each data segment and the order of each compressed segment in the first compressed data, the order of any data segment in the data group is the same as the order of the compressed segment obtained by compressing any data segment in the first compressed data.

[0010] Since the order of any data segment in the data group is the same as the order of the compressed segments obtained by compressing any data segment in the first compressed data, the order of each data segment in the data group can be determined according to the order of each compressed segment in the first compressed data, and then according to the length of each data segment and the order of each data segment in the data group, the initial position of each data segment in the data group can be accurately determined.

[0011] In a possible implementation, determining the starting position of the first compressed segment corresponding to the first data segment according to the starting position of the first data segment includes: obtaining mapping information, the mapping information is used to indicate the mapping relationship between the starting positions of multiple data segments and the starting positions of multiple compressed segments; determining the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment. Since the mapping information can indicate the mapping relationship between the starting positions of multiple data segments and the starting positions of multiple compressed segments, the first data segment belongs to multiple data segments, and thus the starting position of the first compressed segment corresponding to the first data segment can be accurately determined according to the mapping information.

[0012] In a possible implementation, the mapping information includes the starting position of at least one reference data segment in a plurality of data segments and the starting position of at least one reference compressed segment compressed by the at least one reference data segment; the starting position of the first compressed segment is determined according to the mapping information and the starting position of the first data segment, including: determining the first reference data segment in at least one reference data segment according to the starting position of the first data segment and the starting position of at least one reference data segment, the first reference data segment being the reference data segment before the first data segment and the closest to the first data segment; and determining the starting position of the first reference compressed segment compressed by the first reference data segment as the starting position of the first compressed segment. Storing the starting position of at least one reference data segment and the starting position of at least one reference compressed segment in the mapping information can reduce the memory space required for the mapping information and reduce resource waste.

[0013] In a possible implementation, before decompressing the second compressed data according to the second starting position and the ending position of the second compressed data to obtain the target data, it also includes: determining the second data segment to which the second position belongs in multiple data segments according to the starting positions of each data segment, the second position being the sum of the length of the first starting position and the target data; determining the starting position of a third data segment adjacent to the second data segment in the starting positions of each data segment, the third data segment being after the second data segment; determining the starting position of the second compressed segment corresponding to the third data segment according to the starting position of the third data segment; and determining the starting position of the second compressed segment as the ending position of the second compressed data.

[0014] Since the sum of the first starting position and the length of the target data can indicate the end position of the target data in the data group, that is, the second position can indicate the end position of the target data in the data group, the second data segment to which the second position belongs is the data segment to which the end position of the target data belongs, and thus the starting position of the third data segment adjacent to and after the second data segment is located after the end position of the target data, and decompression is stopped from the starting position of the second compressed segment corresponding to the third data segment, which can ensure that the target data is completely decompressed.

[0015] In a possible implementation, decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain the target data includes: decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain intermediate data; determining the end position of the target data according to the first starting position and the length of the target data; and intercepting the target data in the intermediate data according to the first starting position and the end position of the target data. Intercepting the target data in the intermediate data according to the first starting position of the target data and the end position of the target data can ensure that the intercepted data is completely consistent with the target data, so that the accuracy of the decompressed target data is higher.

[0016] In a possible implementation, a data group is stored in a heap-organized table (HOT), the data group includes multiple data in a storage block of the HOT, and the target data includes at least one of the multiple data. The target data is at least one of the multiple data, that is, the target data is part of the data in the data group. Therefore, according to the data processing method provided by the present application, it is possible to partially decompress the compressed data obtained by compressing multiple data to obtain at least one data, thereby ensuring the accuracy of decompression.

[0017] In a possible implementation, a data group is stored in a baseline database of a log structure mergetree (LSM-Tree), the data group includes data in multiple storage blocks in the baseline database, and the target data includes data in at least one storage block among the multiple storage blocks. The target data is data in at least one storage block among the multiple storage blocks, that is, the target data is part of the data in the data group, and therefore, according to the data processing method provided by the present application, it is possible to partially decompress the compressed data obtained by compressing the data in the multiple storage blocks to obtain data in at least one storage block, thereby ensuring the accuracy of the decompression.

[0018] In a second aspect, a data processing method is provided, the method comprising: acquiring a data group, the data group comprising multiple bytes; dividing the multiple bytes into multiple data segments, the multiple data segments comprising at least one of a matching data segment or a non-matching data segment, the bytes in the non-matching data segment are different from the bytes in each data segment before the non-matching data segment, the bytes in the matching data segment are the same as the bytes in a matched data segment before the matching data segment, and the distance between the matching data segment and the matched data segment is less than or equal to a reference distance; compressing the multiple data segments according to the data segments referenced by the multiple data segments to obtain first compressed data comprising multiple compressed segments, each compressed segment being compressed by the corresponding data segment, the data segment referenced by any matching data segment is the matched data segment having the same bytes as those in any matching data segment, and the data segment referenced by any non-matching data segment is any non-matching data segment.

[0019] In the present application, by controlling the distance between the matching data segment and the matched data segment in the data group to be less than or equal to the reference distance, during decompression, the decompression of each compressed segment that needs to be decompressed can be achieved based on the compressed segments corresponding to the data segments whose distances between each data segment do not exceed the reference distance, thereby enabling the clear boundaries of the compressed data that needs to be decompressed to be determined according to the reference distance before decompression, thereby achieving precise decompression.

[0020] In one possible implementation, multiple bytes are divided into multiple data segments, including: dividing the multiple bytes into multiple alternative data segments, the multiple alternative data segments include at least one of alternative matching data segments or non-matching data segments, and the bytes in the alternative matching data segments are the same as the bytes in at least one alternative matched data segment before the alternative matching data segment; obtaining a position array of the alternative matching data segment, the position array including the position of the alternative matching data segment and the position of at least one alternative matched data segment corresponding to the alternative matching data segment; in the case that there is an alternative distance less than or equal to a reference distance in at least one alternative distance, determining the alternative matching data segment as a matching data segment, and at least one alternative distance is determined based on the position of the alternative matching data segment and the position of at least one alternative matched data segment; in the case that at least one alternative distance is greater than the reference distance, determining the alternative matching data segment as a non-matching data segment.

[0021] In the case where at least one candidate distance corresponding to the candidate matching data segment is greater than the reference distance, if the candidate matching data segment is compressed by referring to the candidate matched data segment corresponding to the candidate matching data segment, the candidate matching data segment cannot be decompressed according to the compressed segment corresponding to the data segment whose distance from the candidate matching data segment does not exceed the reference distance. Therefore, in the case where there is a candidate distance less than or equal to the reference distance among at least one candidate distance corresponding to the candidate matching data segment, the candidate matching data segment is determined as the matching data segment, and then the matching data segment is compressed by referring to the matched data segment whose distance from the matching data segment is less than or equal to the reference distance, so that the compressed segment corresponding to the data segment whose distance from the matching data segment does not exceed the reference distance can be decompressed to obtain the matching data segment, thereby ensuring the accuracy of decompression.

[0022] In a possible implementation, after compressing the multiple data segments according to the data segments referenced by the multiple data segments, the method further includes: generating mapping information, the mapping information being used to indicate the mapping relationship between the starting positions of the multiple data segments and the starting positions of the multiple compressed segments. Since the mapping information can indicate the mapping relationship between the starting positions of the multiple data segments and the starting positions of the multiple compressed segments, the starting position of the compressed segment corresponding to the starting position of any data segment can be efficiently determined according to the mapping information during the decompression process, or the starting position of the data segment corresponding to the starting position of any compressed segment can be efficiently determined.

[0023] In a possible implementation, a data group is stored in the HOT, and the data group includes multiple data in a storage block of the HOT. Through the data processing method provided by the present application, multiple data in the storage block are compressed, and the distance between the matching data segment and the matched data segment in the multiple data in the storage block can be controlled to be less than or equal to the reference distance, so that when the compressed data obtained by compressing the multiple data is decompressed, a clear decompression boundary can be determined based on the reference distance to ensure the accuracy of the decompression.

[0024] In a possible implementation, a data group is stored in a baseline database of an LSM-Tree, and the data group includes data in multiple storage blocks in the baseline database. Through the data processing method provided by the present application, the data in multiple storage blocks are compressed, and the distance between the matching data segment and the matched data segment in the data in the multiple storage blocks can be controlled to be less than or equal to the reference distance, so that when the compressed data obtained by compressing the data in the multiple storage blocks is decompressed, a clear decompression boundary can be determined based on the reference distance, thereby ensuring the accuracy of the decompression.

[0025] According to a third aspect, a data processing device is provided, the device comprising: an acquisition module, used to acquire a data processing instruction, the data processing instruction instructs to decompress target data from first compressed data, the first compressed data is obtained by compressing a data group to which the target data belongs, the data processing instruction comprises a first starting position of the target data in the data group and a length of the target data; a determination module, used to determine a second starting position of second compressed data in the first compressed data based on the first starting position and a reference distance, the second compressed data is used to decompress the target data, the reference distance indicates a maximum value of a distance between any data segment in the target data and a data segment referenced by any data segment; a decompression module, used to decompress the second compressed data according to the second starting position and an end position of the second compressed data to obtain the target data, the end position of the second compressed data being determined based on the first starting position and the length of the target data.

[0026] In a possible implementation, the first compressed data includes multiple compressed segments, the data group includes multiple data segments, and each compressed segment is compressed by the corresponding data segment; a determination module is used to obtain the starting position of each data segment in the data group; according to the starting position of each data segment, determine the first data segment to which the first position belongs among the multiple data segments, and the first position is determined based on the difference between the first starting position and a reference distance; according to the starting position of the first data segment, determine the starting position of the first compressed segment corresponding to the first data segment; and determine the starting position of the first compressed segment as the second starting position.

[0027] In one possible implementation, a determination module is used to obtain the length of each data segment; based on the length of each data segment and the order of each compressed segment in the first compressed data, the starting position of each data segment in the data group is determined, and the order of any data segment in the data group is the same as the order of the compressed segment obtained by compressing any data segment in the first compressed data.

[0028] In one possible implementation, a determination module is used to obtain mapping information, where the mapping information is used to indicate a mapping relationship between starting positions of multiple data segments and starting positions of multiple compressed segments; and determine the starting position of the first compressed segment based on the mapping information and the starting position of the first data segment.

[0029] In one possible implementation, the mapping information includes the starting position of at least one reference data segment in multiple data segments and the starting position of at least one reference compressed segment compressed by at least one reference data segment; a determination module is used to determine a first reference data segment in at least one reference data segment based on the starting position of the first data segment and the starting position of at least one reference data segment, the first reference data segment being a reference data segment that is before the first data segment and is closest to the first data segment; and the starting position of the first reference compressed segment obtained by compressing the first reference data segment is determined as the starting position of the first compressed segment.

[0030] In a possible implementation, the determination module is also used to determine, in multiple data segments, the second data segment to which the second position belongs, based on the starting positions of each data segment, the second position being the sum of the first starting position and the length of the target data; determine, in the starting positions of each data segment, the starting position of a third data segment adjacent to the second data segment, the third data segment being after the second data segment; determine, based on the starting position of the third data segment, the starting position of a second compressed segment corresponding to the third data segment; and determine the starting position of the second compressed segment as the end position of the second compressed data.

[0031] In one possible implementation, a decompression module is used to decompress the second compressed data according to the second starting position and the ending position of the second compressed data to obtain intermediate data; determine the ending position of the target data according to the first starting position and the length of the target data; and intercept the target data in the intermediate data according to the first starting position and the ending position of the target data.

[0032] In a possible implementation, the data group is stored in the HOT, the data group includes multiple data in a storage block of the HOT, and the target data includes at least one data among the multiple data.

[0033] In a possible implementation, the data group is stored in a baseline database of the LSM-Tree, the data group includes data in multiple storage blocks in the baseline database, and the target data includes data in at least one storage block among the multiple storage blocks.

[0034] In a fourth aspect, a data processing device is provided, the device comprising: an acquisition module, used to acquire a data group, the data group comprising multiple bytes; a division module, used to divide the multiple bytes into multiple data segments, the multiple data segments comprising at least one of a matching data segment or a non-matching data segment, the bytes in the non-matching data segment are different from the bytes in each data segment before the non-matching data segment, the bytes in the matching data segment are the same as the bytes in a matched data segment before the matching data segment, and the distance between the matching data segment and the matched data segment is less than or equal to a reference distance; a compression module, used to compress the multiple data segments according to the data segments referenced by the multiple data segments to obtain first compressed data including multiple compressed segments, each compressed segment is compressed by the corresponding data segment, the data segment referenced by any matching data segment is the matched data segment having the same bytes as those in any matching data segment, and the data segment referenced by any non-matching data segment is any non-matching data segment.

[0035] In one possible implementation, a partitioning module is used to partition a plurality of bytes into a plurality of alternative data segments, the plurality of alternative data segments including at least one of an alternative matching data segment or a non-matching data segment, the bytes in the alternative matching data segment being the same as the bytes in at least one alternative matched data segment preceding the alternative matching data segment; obtaining a position array of the alternative matching data segment, the position array including the position of the alternative matching data segment and the position of at least one alternative matched data segment corresponding to the alternative matching data segment; determining the alternative matching data segment as a matching data segment when there is an alternative distance less than or equal to a reference distance in at least one alternative distance, the at least one alternative distance being determined based on the position of the alternative matching data segment and the position of at least one alternative matched data segment; determining the alternative matching data segment as a non-matching data segment when at least one alternative distance is greater than the reference distance.

[0036] In a possible implementation, the device further includes a generation module; the generation module is used to generate mapping information, where the mapping information is used to indicate a mapping relationship between the starting positions of the multiple data segments and the starting positions of the multiple compressed segments.

[0037] In a possible implementation, a data group is stored in a HOT, and the data group includes a plurality of data in a storage block of the HOT.

[0038] In a possible implementation, the data group is stored in a baseline database of the LSM-Tree, and the data group includes data in multiple storage blocks in the baseline database.

[0039] In a fifth aspect, the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the data processing method provided by the aforementioned first aspect, second aspect, and any possible implementation of the first aspect or the second aspect.

[0040] In a sixth aspect, an embodiment of the present application provides a computer program product including instructions, which, when executed by a computing device cluster, enables the computing device cluster to perform a data processing method as provided in the first aspect, the second aspect, and any possible implementation of the first aspect or the second aspect. The computer program product may be a software installation package, which may be downloaded and executed on the computing device cluster when the functions of the computing device cluster are to be implemented.

[0041] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method provided by the first aspect, the second aspect, and any possible implementation of the first aspect or the second aspect. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0042] It should be understood that the beneficial effects achieved by the technical solutions of the third to seventh aspects of the present application and the corresponding possible implementation methods can be referred to the technical effects of the data processing method provided for the first aspect, the second aspect and any possible implementation method of the first aspect or the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a general data compression process based on the LZ77 algorithm provided for related technologies;

[0044] Figure 2 A schematic diagram of a general data compression process based on the LZ77 algorithm provided for related technologies;

[0045] Figure 3 A schematic diagram of an implementation scenario of a data processing method provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of a data compression method provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a HOT-based data processing environment provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a data processing environment based on LSM-Tree provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of mapping information provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of a data compression process provided by an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of a data decompression process provided in an embodiment of the present application;

[0052] Fig.10A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0055] Fig.13 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0056] Fig.14 A schematic diagram of a connection method for a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0058] With the development of cloud computing technology, cloud services provided based on cloud computing technology are becoming more and more diverse. For complex calculations performed to provide cloud services, a large amount of data needs to be called during the calculation process, and therefore a large amount of data needs to be stored. For example, data management systems such as databases are used to store a large amount of data corresponding to a variety of cloud services, and with the update and development of cloud services, the amount of data that the data management system needs to store will increase accordingly. However, in order to store the increasing amount of data, the data management system needs to continuously expand the system capacity, which consumes a lot of storage costs, and system expansion of the data management system will cause the data management system to be unstable, resulting in problems such as errors or loss of stored data.

[0059] Therefore, when storing data, data compression technology is usually used to compress the data. The compressed data volume is reduced. Storing compressed data can reduce the storage cost of data and reduce the frequency and possible risks of system expansion. For example, a group of data can be compressed as a whole using data compression technology. For example, a group of user records in a database can be compressed as a whole. Among them, the data compression technology can be LZ77 (a data compression technology), Huffman compression technology, etc. LZ77 is a lossless data compression algorithm. Huffman compression technology achieves data compression by encoding data.

[0060] In the process of calculation, when it is necessary to calculate the compressed data, it is necessary to first query the compressed data required for the calculation, decompress the compressed data required for the calculation, obtain the decompressed data, and complete the calculation based on the decompressed data. Since the compressed data may be partially queried during the calculation process, that is, the data to be queried is part of the compressed data, it is not necessary to decompress all the compressed data, and the compressed part of the data can be decompressed, or the compressed data can be partially decompressed to obtain the required part of the data.

[0061] Since different compressed data are obtained by compressing data based on different data compression technologies, different partially decompressed data are used in different ways. For example, in a related art, data compression technologies such as dictionary encoding or run length encoding (RLE) are used to compress data according to the characteristics of the data to be compressed, so that the compressed data can support partial decompression.

[0062] Taking dictionary encoding as an example of related technology 1, related technology 1 can divide the data to be compressed into multiple independent fields according to business needs and business characteristics, find repeated fields with higher frequency in multiple fields, store the values ​​of repeated fields in an independent data dictionary, and replace the values ​​of repeated fields in the data to be compressed with the positions of the repeated fields in the data dictionary to obtain compressed data, thereby completing data compression. When decompressing the compressed data, data can be restored according to the positions of the repeated fields in the compressed data in the data dictionary and the values ​​in the data dictionary to complete data decompression. When part of the data needs to be decompressed, the positions in the data dictionary included in the part of the data need to be restored to the values ​​of the corresponding repeated fields to obtain the restored data, thereby achieving decompression of part of the data.

[0063] However, the related technology 1 can only identify repeated fields in the data, but cannot identify redundant repeated information within the fields, and thus cannot compress the unidentified repeated information, making it difficult to ensure the compression rate of repeated information in the data, and further making it difficult to ensure the stability of the compression rate. In addition, the related technology 1 needs to divide the data according to business needs and business characteristics, but in some cases, it is difficult to divide the data according to business needs and business characteristics, and the related technology 1 is difficult to apply to the compression and decompression scenarios in such situations, so the scenario adaptability of the technology is poor.

[0064] The data compression field also provides another related technology 2, which divides the data to be compressed into multiple independent parts, and compresses each part separately to obtain multiple compressed segments. When part of the data needs to be decompressed, the compressed segment where the part of the data to be decompressed is located can be decompressed.

[0065] However, since repeated information in the data may be located in different parts, data compression through related technology 2 makes it impossible to identify and eliminate repeated information located in different parts, thereby reducing the overall compression rate of the data. Exemplarily, when the data to be compressed is small in size, for example, the data to be compressed is data in a page of a database, and the length of data in a page is usually 8 kilobytes (KB) to 16KB. The repetition rate of smaller-scale data itself may be low, and dividing the data into multiple independent parts may result in a lower repetition rate of information in each independent part, or even no repetition of information in each independent part, resulting in a lower compression rate.

[0066] In addition, the related technology 2 compresses multiple parts of data separately through independent compression processes, and needs to call the compression algorithm multiple times. That is, each time an independent part is compressed, the compression algorithm needs to be called. When calling the compression algorithm, it may be necessary to introduce an index structure for compression, and independent index structures need to be maintained for different parts to identify repeated information in different parts. Calling and starting the compression algorithm multiple times will trigger the initialization of the index structure, making the startup cost of calling the compression algorithm high, resulting in a waste of resources.

[0067] The present application provides a data processing method that can compress and decompress data and support decompression of part of the data. The compression rate of the method is stable, the scene adaptability is strong, and the startup cost of calling the compression algorithm is small, which can avoid waste of resources. The data processing method can be implemented based on a general compression algorithm, for example, based on the LZ77 algorithm.

[0068] In order to make the description of the data processing method provided in the embodiment of the present application easier to understand, before describing the data processing method provided in the embodiment of the present application, the process of general data compression and decompression based on the LZ77 algorithm is first schematically described.

[0069] For example, see Figure 1 , shows a schematic diagram of a general data compression process based on the LZ77 algorithm provided by the related art. First, the data to be compressed is obtained, and the data to be compressed is regarded as a byte stream including multiple bytes, and the multiple bytes are processed in sequence. Before processing each byte, it is determined whether the processing of all bytes in the byte stream has been completed. If there are still bytes that have not been processed, the next byte is processed.

[0070] Taking the processing of byte x as an example, the matched data segment of byte x must be searched first. For example, the byte before byte x can be searched for the byte that is the same as byte x. If the byte before byte x contains the byte that is the same as byte x, byte x is temporarily recorded as the matched data segment, and the byte before byte x that is the same as byte x is temporarily recorded as the matched data segment. If there is no byte that is the same as byte x, byte x is temporarily recorded as the non-matched data segment. The matched data segment can also be called the matched string, the matching data segment can also be called the matching string, and the non-matching data segment can also be called the non-matching string.

[0071] For example, see Figure 2 , shows a schematic diagram of a general data compression process based on the LZ77 algorithm provided by the related art. The acquired data to be compressed is ABCDEFBCDGDEFEAFBC, each letter is one byte, so the data to be compressed can be regarded as a byte stream containing 18 bytes. Since the data to be compressed is in text form, the data to be compressed can also be called the text to be compressed or the original text.

[0072] After compression begins, each byte in the byte stream is processed in turn. When processing the first byte A, first find the matched data segment of the first byte A. Since the first byte A is the first byte in the byte stream, there is no byte identical to the first byte A before the first byte A, that is, the matched data segment corresponding to the first byte A cannot be found in the byte stream, so the first byte A can be temporarily recorded as a non-matching data segment. The second byte B, the third byte C, the fourth byte D, the fifth byte E, and the sixth byte F cannot find the corresponding matched data segment in the byte stream, so the second byte B, the third byte C, the fourth byte D, the fifth byte E, and the sixth byte F can be temporarily recorded as non-matching data segments. When processing the seventh byte B, it is found that the second byte B is identical to the seventh byte B, and the second byte B is before the seventh byte B, so the seventh byte B can be temporarily recorded as a matching data segment, and the second byte B can be temporarily recorded as the matched data segment of the seventh byte B. When processing the eighth byte C and the ninth byte D, it is found that the third byte C is the same as the eighth byte C, and the fourth byte D is the same as the ninth byte D. Therefore, the eighth byte C and the ninth byte D can be temporarily recorded as matching data segments, the third byte C can be temporarily recorded as the matched data segment of the eighth byte C, and the fourth byte D can be temporarily recorded as the matched data segment of the ninth byte D.

[0073] In the process of sequentially processing each byte in the byte stream, if several consecutive bytes are the same as several previous consecutive bytes, the several consecutive bytes can be merged into a matching data segment, and the several previous consecutive bytes are determined as the matched data segment of the matching data segment. One or more consecutive bytes temporarily recorded as non-matching data segments between two adjacent matching data segments can be merged into a non-matching data segment.

[0074] For example, see Figure 2 When processing the tenth byte G, the byte identical to the tenth byte G cannot be found in the byte stream, so the tenth byte G can be temporarily recorded as a non-matching data segment. The seventh consecutive byte B, the eighth byte C, and the ninth byte D are identical to the second consecutive byte B, the third byte C, and the fourth byte D, while the tenth byte G is different from the fifth byte F. Therefore, the seventh consecutive byte B, the eighth byte C, and the ninth byte D can be combined into a matching data segment BCD, and the second consecutive byte B, the third byte C, and the fourth byte D can be determined as the matched data segment BCD of the matching data segment.

[0075] In addition, since the first byte A to the sixth byte F are temporarily recorded as non-matching data segments, and the seventh byte B after the sixth byte F has been determined to belong to the matching data segment, the first byte A to the sixth byte F can be combined into a non-matching data segment ABCDEF. Figure 2 The processing process of other bytes in the byte stream in can refer to the above description and will not be repeated here.

[0076] In some cases, after recording and merging the matching data segments and the non-matching data segments, the final matching data segments and the non-matching data segments can be determined based on the lengths of the matching data segments after merging. If the length of the matching data segment after merging is less than the length threshold, the matching data segment can be re-determined as a non-matching data segment.

[0077] for example, Figure 2The fourteenth byte E in is the same as the twelfth byte E and the fifth byte E, and the fourteenth byte E is recorded as the matching data segment E, but the length of the matching data segment E is 1 byte (byte, B), which is less than the length threshold 3B, so the matching data segment E can be re-determined as the non-matching data segment E. The fifteenth byte A is the same as the first byte A, and the fifteenth byte A is recorded as the matching data segment A, but the length of the matching data segment A is 1B, which is less than the length threshold 3B, so the matching data segment A can be re-determined as the non-matching data segment A. Since the byte F before the fourteenth byte E belongs to the matching data segment DEF whose length is equal to the length threshold, and the byte F after the fifteenth byte A also belongs to the matching data segment FBC whose length is equal to the length threshold, the fourteenth byte E and the fifteenth byte A can be combined into a non-matching data segment EA.

[0078] After the final matching data segments and non-matching data segments are determined, data compression can be performed based on the characteristics of each matching data segment and non-matching data segment, and the compressed data can be output. For non-matching data segments, the lengths of the non-matching data segments and non-matching data segments are output; for matching data segments, the length of the matching data segments and the matching displacement of the matching data segments are output. The matching displacement of the matching data segment refers to the distance between the matching data segment and the matched data segment of the matching data segment, for example, it can be the length of bytes between the first byte in the matching data segment and the last byte in the matched data segment.

[0079] Continue with Figure 2Take the byte stream in as an example, the first byte A to the sixth byte F is finally determined to be a non-matching data segment ABCDEF with a length of 6B, then the length 6B and the non-matching data segment ABCDEF can be output; the seventh byte B to the ninth byte D are finally determined to be a matching data segment BCD with a length of 3B, and the matched data segment BCD of the matching data segment BCD is the second byte B to the fourth byte D, and the distance between the matching data segment BCD and the matched data segment BCD is 2B, that is, the matching displacement of the matching data segment BCD is 2B, then the length 3B and the matching displacement 2B can be output (the unit B of the length and displacement can be ignored); the tenth byte G is finally determined to be a non-matching data segment G with a length of 1B, then the length 1B and the non-matching data segment G can be output; the eleventh byte D to the thirteenth byte F are finally determined to be a matching data segment with a length of 3B. According to the embodiment of the present invention, the matched data segment DEF of the matching data segment DEF is determined to be the matched data segment DEF of the fourth byte D to the sixth byte F, and the distance between the matching data segment DEF and the matched data segment DEF is 4B, that is, the matching displacement of the matching data segment DEF is 4B, and the length 3B and the matching displacement 4B can be output; the fourteenth byte E to the fifteenth byte A are finally determined to be a non-matching data segment EA with a length of 2B, and the length 2B and the non-matching data segment EA can be output; the sixteenth byte F to the eighteenth byte C are finally determined to be a matching data segment FBC with a length of 3B, and the matched data segment FBC of the matching data segment FBC is the sixth byte F to the eighth byte C, and the distance between the matching data segment FBC and the matched data segment FBC is 7B, that is, the matching displacement of the matching data segment is 7B, and the length 3B and the matching displacement 7B can be output.

[0080] Among them, since the process of compressing the matching data segment needs to be based on the position of the matched data segment and the distance between the matching data segment and the matched data segment, in some cases, it can be considered that the matching data segment is compressed by referencing the matched data segment. Correspondingly, in the process of compressing the non-matching data segment, it is necessary to compress based on the value of the non-matching data segment and the length of the non-matching data segment, so it can also be considered that the non-matching data segment is compressed by referencing the non-matching data segment itself.

[0081] After determining the output for each matching data segment and non-matching data segment, the output results can be sorted to obtain compressed data. The compressed data is also in text form, so the compressed data can also be called compressed text. Figure 2 , Figure 2The compressed data obtained by the above compression method is: (6) ABCDEF (2, 3, 1) G (4, 3, 2) EA (7, 3). Among them, 6 is the length 6B of the non-matching data segment ABCDEF; 2, 3 and 1 respectively represent the matching displacement 2B of the matching data segment BCD, the length 3B of the matching data segment BCD and the length 1B of the next non-matching data segment G adjacent to the matching data segment BCD; 4, 3 and 2 respectively represent the matching displacement 4B of the matching data segment DEF, the length 3B of the matching data segment DEF and the length 2B of the next non-matching data segment EA adjacent to the matching data segment DEF; 7 and 3 respectively represent the matching displacement 7B of the matching data segment FBC and the length 3B of the matching data segment FBC. Since there is no non-matching data segment after the matching data segment FBC, the length of the non-matching data segment after the matching data segment FBC can be omitted.

[0082] After completing data compression based on the above compression method, the data system will store the compressed data and no longer store the data before compression, so as to reduce the space required for storing data and save storage costs. Afterwards, if the stored data needs to be called, the stored data can be decompressed first to obtain the decompressed data for data calculation. If the data to be called is part of the compressed data, it is necessary to partially decompress the part of the data and restore the part of the data before compression.

[0083] For example, see Figure 2 , the data system stores the compressed data (6)ABCDEF(2,3,1)G(4,3,2)EA(7,3). If the partial data to be called is DEFEAFBC in the original data, the caller sends the starting position of the partial data in the complete pre-compressed data and the length of the partial data to the data management system. Optionally, the starting position of the partial data in the complete pre-compressed data can be the position of the first byte in the partial data in the complete pre-compressed data. The position of each byte in the complete pre-compressed data can be the number of bytes between each byte and the first byte in the complete pre-compressed data (for example Figure 2 For example, the first byte D in the partial data DEFEAFBC is the eleventh byte in the complete data before compression, so the starting position of the partial data DEFEAFBC in the complete data before compression is 10. The length of the partial data may be the length of the bytes of the partial data, for example, if the number of bytes of the partial data DEFEAFBC is 8 and the length of each byte is 1B, the length of the partial data DEFEAFBC is 8B.

[0084] After receiving the starting position of the partial data in the complete data before compression and the length of the partial data, the data management system determines the partial compressed data corresponding to the partial data in the compressed data, and decompresses the partial compressed data to obtain the partial data before compression. The data management system can determine the partial compressed data corresponding to the partial data according to the length and order of each data segment contained in each compressed segment in the compressed data and the starting position and length of the partial data.

[0085] For example, based on the above example, the first compressed segment (6) ABCDEF in the compressed data includes the length 6B of the first data segment ABCDEF, the second compressed segment (2, 3, 1) includes the length 3B of the second data segment BCD and the length 1B of the third data segment G, the fourth compressed segment (4, 3, 2) includes the length 3B of the fourth data segment DEF and the length 2B of the fifth data segment EA, and the sixth compressed segment (7, 3) includes the length 3B of the sixth data segment FBC.

[0086] If the starting position of the partial data DEFEAFBC to be called is 10 in the complete data before compression, it is necessary to find the compressed segment (4, 3, 2) corresponding to the data segment DEF to which the eleventh byte D belongs. The length of the first data segment ABCDEF is 6B, so the eleventh byte D does not belong to the first data segment ABCDEF; the cumulative length of the first data segment ABCDEF and the second data segment BCD is 9B, so the eleventh byte D does not belong to the second data segment BCD; the cumulative length from the first data segment ABCDEF to the third data segment G is 10B, so the eleventh byte D does not belong to the third data segment G, but it can be determined that the eleventh byte D belongs to the data segment after the third data segment G, that is, the fourth data segment DEF, so it can be determined that it is necessary to start decompression from the fourth compressed segment (4, 3, 2) corresponding to the fourth data segment DEF.

[0087] After determining the compression segment to which the starting position of the partial compressed data to be decompressed belongs, the compression segment to which the ending position of the partial compressed data to be decompressed belongs can be determined based on the starting position and length of the partial data. For example, the starting position of the partial data DEFEAFBC to be called in the entire original data is 10, and the length of the partial data to be called is 8B, so it can be determined that the ending position of the partial data DEFEAFBC to be called in the complete pre-compression data is 17. The cumulative length of the first data segment ABCDEF to the sixth data segment FBC in the complete pre-compression data is 18B, so it can be determined that the compression segment to which the ending position of the partial data DEFEAFBC to be called belongs is the sixth compression segment (7, 3) corresponding to the sixth data segment FBC, and the compressed segments to be decompressed are the fourth compression segment (4, 3, 2) to the sixth compression segment (7, 3).

[0088] After determining the compressed segments that need to be decompressed, the compressed segments that need to be decompressed can be decompressed in sequence. The compressed segment obtained by compressing the non-matching data segment can be decompressed to obtain the value of the data segment before compression of the compressed segment according to the length of the compressed segment and the value of the compressed segment. The compressed segment obtained by compressing the matching data segment can be decompressed to obtain the value of the data segment before compression of the compressed segment according to the length of the matching data segment in the compressed segment, the distance between the matching data segment and the referenced matched data segment, and the value of the matched data segment.

[0089] Continuing with the above example, the compressed segments that need to be decompressed are determined to be the fourth compressed segment (4, 3, 2) to the sixth compressed segment (7, 3), and the fourth compressed segment (4, 3, 2) can be decompressed first. According to the format of the value of the fourth compressed segment (4, 3, 2), it can be known that the fourth data segment compressed to obtain the fourth compressed segment is a matching data segment, and thus it can be determined according to the value in the fourth compressed segment (4, 3, 2) that the distance between the matched data segment referenced when compressing the fourth compressed segment (4, 3, 2) and the fourth data segment DEF is 4B, and further, the compressed segment corresponding to the data segment with a distance of 4B from the fourth data segment DEF can be determined in the compressed data before the fourth compressed segment (4, 3, 2).

[0090] By reading the value of the second compressed segment (2, 3, 1), we know that the length of the third data segment G corresponding to the third compressed segment G is 1B, the length of the second data segment BCD corresponding to the second compressed segment (2, 3, 1) is 3B, and the sum of the lengths of the second data segment BCD and the third data segment G is 4B, indicating that the matched data segment referenced by the fourth data segment DEF is the data segment before the second data segment BCD, that is, the first data segment ABCDEF, and the fourth data segment BCD references from the last byte F of the first data segment ABCDEF, and references three bytes forward in total. Since the first data segment ABCDEF obtained by compression is a non-matching data segment, the last byte F, the second to last byte E and the third to last byte D in the first compressed segment (6) ABCDEF can be directly read to output the data segment DEF, and the decompression and restoration of the fourth compressed segment DEF is completed.

[0091] Since the data segment obtained by compression of the fifth compressed segment EA is a non-matching data segment, the value EA of the fifth compressed segment can be directly read, and the data segment EA can be output to complete the decompression and restoration of the fifth data segment EA. According to the format of the value of the sixth compressed segment (7, 3), the sixth data segment FBC obtained by compression of the sixth compressed segment (7, 3) is a matching data segment, and thus the distance between the matched data segment referenced when compressing the sixth compressed segment (7, 3) and the sixth data segment FBC can be determined to be 7B according to the value in the sixth compressed segment (7, 3), and thus the data segment with a distance of 7B from the sixth data segment FBC can be determined in the compressed data before the sixth compressed segment (7, 3).

[0092] By reading the values ​​of the second compressed segment (2, 3, 1) and the fourth compressed segment (4, 3, 2), it can be known that the length of the fourth data segment DEF corresponding to the fourth compressed segment (4, 3, 2) is 3B, the length of the fifth data segment EA corresponding to the fifth compressed segment EA is 2B, the length of the third data segment G corresponding to the third compressed segment G is 1B, and the sum of the lengths of the fifth data segment EA to the third data segment G is 6B, indicating that the matched data segment referenced by the sixth data segment FBC is the data segment before the third data segment G, that is, the second data segment BCD, and the sixth data segment FBC references from the second to last byte C of the second data segment BCD, and references three bytes forward in total. Since the second data segment BCD of the second compressed segment (2, 3, 1) obtained by compression is a matching data segment, it is necessary to first decompress the second compressed segment (2, 3, 1) and restore it to the second data segment BCD, and then decompress the sixth compressed segment (7, 3) according to the second data segment BCD.

[0093] That is, the matched data segment directly referenced in the process of compressing the sixth data segment FBC includes bytes in the second data segment BCD, and the first data segment ABCDEF is directly referenced in the process of compressing the second data segment BCD, resulting in the sixth data segment FBC indirectly referencing the first data segment ABCDEF, and thus two layers of decompression are required in the process of decompressing the sixth compressed segment (7, 3).

[0094] Combined with the above examples, it can be seen that when partially decompressing the compressed data obtained by the general data compression method based on LZ77, there will be a situation where two layers of decompression or even multiple layers of decompression are required, and the situation where double-layer decompression or multi-layer decompression is required cannot be predicted before decompression, that is, before the decompression is completed, the number, size and position of the compressed segments or data segments actually involved in the decompression cannot be determined, and the decompression efficiency is positively correlated with the number and size of the compressed segments that need to be decompressed. Therefore, it is difficult to guarantee the efficiency of decompression when the number and size of the compressed segments or data segments actually involved in the decompression are uncertain. For scenes that require frequent partial decompression, there is no stable decompression boundary and decompression efficiency, which will lead to unstable decompression performance. Among them, the decompression performance can be determined by combining the decompression efficiency and the accuracy of decompression. If there are differences in decompression performance for different data, it may affect the operation of the business that needs to query data, and it is difficult to guarantee the user experience.

[0095] The data processing method provided in the embodiment of the present application can ensure that the decompression boundary is determined before the data is decompressed, thereby stabilizing the decompression performance. Figure 3 , shows an implementation scenario of the method, which includes a device 11 for providing cloud computing services. Optionally, the device 11 can be deployed in the cloud, for example, the device 11 can be a data management device in the cloud or other devices in the cloud that need to process data. Different services 12 and data systems 13 can be deployed in the device 11, and the device 11 can store and process data through the data system 13. The data system 13 can also be called a data management system, and the data system 13 can be a system for managing data such as a database, and a data processing environment based on HOT and a data processing environment based on LSM-Tree can be installed in the data system 13.

[0096] The data processing method provided in the embodiment of the present application includes a data compression method and a data decompression method. The data compression method can provide a basis for stabilizing the performance of data decompression. The data compression method provided in the embodiment of the present application is described below. Figure 4 The flowchart of the data compression method is shown, and the data compression method includes but is not limited to the following S401 to S403.

[0097] S401, obtaining a data group, where the data group includes a plurality of bytes.

[0098] Wherein, a data group is a compression unit of data. A data group may include one or more groups of data, a group of data may include one or more data, a data may include one or more bytes, and thus a data group may include one or more bytes. Since if a data group includes one byte, there is no need to compress the data group, and the embodiment of the present application is intended to compress or decompress the data group, the data processing method in the case where the data group only includes one byte is not discussed in the embodiment of the present application, but the data processing method and process are described by taking the case where the data group includes multiple bytes as an example.

[0099] The embodiment of the present application does not limit the size of the data group, and the size of the data group in different data processing environments can be the same or different. Below, the embodiment of the present application takes the data processing environment based on HOT and the data processing environment based on LSM-Tree as examples to illustrate the composition and determination process of the data group.

[0100] For example, see Figure 5 , showing a schematic diagram of a data processing environment based on HOT. In a data processing environment based on HOT, data is stored in an independent heap located in the HOT. The heap is a logical collection of multiple data. Each data stored in the heap corresponds to an address for obtaining the data and a key value (Key, K) for querying the data. The address for obtaining the data can be represented by a row identifier (row identifier, RID). Based on HOT, a binary (binary, B) +-tree index can be established based on the Key of each data. The bottom node of the index contains multiple tuples. A tuple includes the Key and RID of a data. Each tuple can be represented as<Key,RID> or<K,R> In a B+-Tree, each tuple is sorted in the order of the included keys. For example, Figure 5 The bottom layer nodes of the B+-Tree in the above example include 6 tuples, which are<K1,R1> ,<K2,R2> ,<K3,R3> ,<K4,R4> ,<K5,R5> and<K6,R6> , the 6 tuples are arranged in order of K from small to large.

[0101] The physical structure of HOT includes multiple storage blocks (Blocks), each of which stores multiple data. The size of each storage block can be, for example, 8KB-16KB. Since different data have different values ​​(value, V), different data can also be represented by different values, so that different data can be distinguished by different values. The order of arrangement of multiple data in each storage block has nothing to do with the K corresponding to the multiple data. For example, Figure 5 The storage block indicated by ① includes data 2 represented by V2, data 3 represented by V3 and data 6 represented by V6. Data 2 corresponds to K2, data 3 corresponds to K3, and data 6 corresponds to K6. The arrangement order of the three data is data 3-data 6-data 2, and they are not arranged in the order of K of each data.

[0102] In a data processing environment based on HOT, multiple data in a storage block that meet a compression condition can be compressed. The multiple data that meet the compression condition can be all the data stored in the storage block or part of the data stored in the storage block. The compression condition can be set based on experience or specified by a user. For example, the compression condition can be a data length threshold. If the length of the data is greater than or equal to the length threshold, the data can be determined as data that meets the compression condition. On the contrary, if the length of the data is less than the length threshold, the data can be determined as data that does not meet the compression condition.

[0103] Multiple data used for compression in a storage block can be called a data group. That is, in the embodiment of the present application, when the data group is stored in the HOT, the data group can include multiple data in the storage block of the HOT. Figure 5 In the compression process indicated by ②, it is determined that data 2 represented by V2 and data 3 represented by V3 are data that meet the compression conditions, then data 2 and data 3 can be regarded as a data group, that is, the data group includes data 2 and data 3 in the storage block.

[0104] For example, see Figure 6 , shows a schematic diagram of a data processing environment based on LSM-Tree. In the storage engine based on LSM-Tree, the stored data includes baseline data and incremental data. The baseline data can be the original data uploaded by the user, and the baseline data is stored in a sorted string table (SSTable). SSTable is as follows: Figure 6 As shown in ①.

[0105] In terms of physical structure, the smallest unit corresponding to an SSTable is a block group (Chunk), which can also be called a baseline database. The size of a baseline database can be 64KB-2MB. A baseline database includes multiple storage blocks, each of which stores one or more baseline data persistently. The size of each storage block can be different, and the size of the storage block can be any size within a reference range. The reference range can be determined based on experience or specified by the user. The reference range can be, for example, 8KB-16KB. Logically, each baseline data is arranged in the order of K corresponding to each baseline data. Among them, K is the value used to locate each baseline data, and Figure 5 The K corresponding to each baseline data is maintained in independent metadata, so the value range distribution of K for each storage block can be recorded through metadata. In addition, each baseline data can be represented by a different value (V). For example, Figure 6 In a storage block of the SSTable shown in ①, baseline data 1 represented by V1 and baseline data 2 represented by V2 are stored. Baseline data 1 corresponds to K1, and baseline data 2 corresponds to K2. Baseline data 1 and baseline data 2 are arranged in ascending order according to their corresponding K1 and K2.

[0106] Another type of data stored in the LSM-Tree based storage engine, namely incremental data, is used to record updates or modifications to baseline data, where modifications to baseline data include but are not limited to insertions or deletions. Figure 6 As shown in ④, the incremental data is stored in the memory table (MemTable). Optionally, the memory table can also be called a metadata table. An incremental data record in the memory table updates or modifies a baseline data, for example, Figure 6 The first incremental data in the memory table in records the insertion of baseline data 3 represented by V3, and baseline data 3 corresponds to K3; the second incremental data records the update of the value of baseline data 5 corresponding to V5 to V5'.

[0107] Since the update or modification of the baseline data does not directly affect the baseline data, when querying the baseline data based on business needs, it is necessary to merge the incremental data with the data required by the business in the baseline data, and return the merged baseline data. For example, when querying the baseline data 5 based on business needs, it is determined that the incremental data corresponding to the baseline data 5 is to update the value of the baseline data 5 from V5 to V5', so the baseline data 5 and the incremental data corresponding to the baseline data 5 can be merged, that is, the value of the baseline data 5 is updated from V5 to V5', to obtain the merged baseline data 5, and return the merged baseline data 5.

[0108] Since the size of MemTable is limited, when MemTable grows to a certain size, it will trigger the automatic merging of incremental data and baseline data (such as Figure 6 ⑤ in the process), generate new baseline data (such as Figure 6 The new baseline data will not directly overwrite the old baseline data, but will be stored in Figure 6 In the new SSTable shown in ⑦, the new SSTable is located in the newly allocated storage space, and the new baseline data is persistently stored in the new SSTable. The baseline data before the merge is the old baseline data, and the SSTable storing the old baseline data is the old SSTable.

[0109] The old baseline data will expire after meeting the expiration condition, and the storage space occupied by the old SSTable will be released. The expiration condition may include a time threshold after the new baseline data is generated. If the time after the new baseline data is generated reaches the time threshold, the old baseline data can be deleted to release the storage space occupied by the old SSTable.

[0110] In an embodiment of the present application, data can be compressed in the process of generating new baseline data based on the merger of old baseline data and incremental data. In some cases, the baseline database is the minimum unit of data writing and space allocation in the process of input / output (I / O), so in the compression process, all baseline data in a baseline database can be compressed as a data group, or a plurality of consecutive modified storage blocks in a baseline database can be compressed as a data group. That is, in the case where the data group is stored in the baseline database of the LSM-Tree, the data group includes data in multiple storage blocks in the baseline database.

[0111] Regardless of the data processing environment, and regardless of the data included in the data group, the data group may include multiple bytes, and the data group including multiple bytes may be compressed. The process of compressing the data group including multiple bytes may refer to S402 and S403 below, which will not be described in detail here.

[0112] S402, divide the multiple bytes into multiple data segments, the multiple data segments include at least one of a matching data segment or a non-matching data segment, the bytes in the non-matching data segment are different from the bytes in each data segment before the non-matching data segment, the bytes in the matching data segment are the same as the bytes in a matched data segment before the matching data segment, and the distance between the matching data segment and the matched data segment is less than or equal to the reference distance.

[0113] Among them, the description of the matching data segment, the non-matching data segment and the matched data segment can refer to the above description of the general compression algorithm based on LZ77, which will not be repeated here. In the process of dividing multiple data segments in the embodiment of the present application, it is necessary to constrain the distance between the matching data segment and the matched data segment so that the distance between the final matching data segment obtained by division and the matched data segment is less than or equal to the reference distance. The size of the reference distance can be set according to experience or pre-defined according to user needs.

[0114] The embodiment of the present application does not limit the method for constraining the distance between the matching data segment and the matched data segment. For example, taking the distance between the matching data segment and the matched data segment constrained by the position array as an example, the process of dividing multiple bytes into multiple data segments may include: dividing multiple bytes into multiple candidate data segments; obtaining the position array of the candidate matching data segment; in the case that there is an alternative distance less than or equal to the reference distance in at least one alternative distance, determining the alternative matching data segment as a matching data segment; in the case that at least one alternative distance is greater than the reference distance, determining the alternative matching data segment as a non-matching data segment. Among them, the multiple alternative data segments include at least one of the alternative matching data segment or the non-matching data segment, and the bytes in the alternative matching data segment are the same as the bytes in at least one alternative matched data segment before the alternative matching data segment. The process of dividing multiple bytes into multiple alternative data segments is the same as the process of dividing multiple bytes into multiple data segments by the above-mentioned general compression algorithm based on LZ77, and will not be repeated here.

[0115] The position array used to constrain the distance between the matching data segment and the matched data segment can be generated in the process of dividing to obtain multiple candidate data segments, and the position array may include the position of the candidate matching data segment and the position of at least one candidate matched data segment corresponding to the candidate matching data segment. The position of the candidate matching data segment includes the position of each byte in the candidate matching data segment, and the position of at least one candidate matched data segment includes the position of each byte in at least one candidate matched data segment. In an embodiment of the present application, the position of each byte can be represented by the offset position of each byte in the byte stream, or by the order of each byte in the byte stream.

[0116] The embodiment of the present application provides a symbol system applied to a data processing method, and the symbol system includes a representation method for each byte, a representation method for each data segment, and a representation method for a position array. Below, the composition and function of the position array are explained through the description of the symbol system. In this symbol system, the data group to be compressed is regarded as a byte stream of length n, and the byte stream is set to S. S[i] represents the i-th byte in S, and S[i..j] represents the data segment starting from the i-th byte and ending at the j-1-th byte in S, that is, the substring from the i-th byte to the j-1-th byte. S[i..j] can also be expressed in the form of S[i, j] or S[ij]. Correspondingly, in this symbol system, the first compressed data after compression is regarded as a byte stream of length m, and the compressed byte stream is set to D. The representation method of each compressed segment in D is the same as the representation method of each data segment in S, which will not be repeated here.

[0117] The length of the position array is n. Each byte in the byte stream corresponds to at least one prototype position. The prototype position of each byte is the position of the byte referenced by each byte. The position array is used to record the correspondence between the position of each byte and the prototype position of each byte, so the position data can also be called the prototype array.

[0118] The prototype position of a byte in a non-matching data segment is the position of the byte itself. For example, the prototype position of the first byte in a non-matching data segment is the position of the first byte itself. In the symbology provided in the embodiment of the present application, for a non-matching data segment S[i..i+l], Ancestor[i+k]=i+k is set in the position array, k is greater than or equal to 0 and k is less than l, and l is the length (length, l) of the candidate matching data segment and the candidate matched data segment.

[0119] The prototype position of the byte in the candidate matching data segment is the prototype position of the corresponding byte in the candidate matched data segment corresponding to the candidate matching data segment. If the candidate matched data segment belongs to the non-matching data segment, that is, the candidate matched data segment does not reference other data segments, then the prototype position of the byte in the candidate matching data segment is the position of the corresponding byte in the candidate matched data segment corresponding to the candidate matching data segment; if the candidate matched data segment belongs to the matching data segment, that is, the candidate matched data segment references other candidate matched data segments, then the prototype position of the byte in the candidate matching data segment may include at least one of the position of the corresponding byte in the candidate matched data segment or the position of the corresponding byte in other candidate matched data segments.

[0120] For example, if the alternative matched data segment referenced by the alternative matched data segment 1 is the alternative matched data segment 2, and the alternative matched data segment 2 belongs to the non-matching data segment, and the byte referenced by the first byte in the alternative matched data segment 1 is the first byte in the alternative matched data segment 2, then the prototype position of the first byte in the alternative matched data segment 1 is the position in the byte stream of the first byte in the alternative matched data segment 2. For another example, if the alternative matched data segment 2 belongs to the alternative matched data segment 4, and the alternative matched data segment 4 references the alternative matched data segment 5, and the alternative matched data segment 5 belongs to the non-matching data segment 6, then the alternative matched data segment 1 directly references the alternative matched data segment 2, and the alternative matched data segment 1 indirectly references the alternative matched data segment 5. The byte directly referenced by the first byte in the candidate matching data segment 1 is the first byte in the candidate matching data segment 2, and the byte indirectly referenced by the first byte in the candidate matching data segment 1 is the first byte in the candidate matching data segment 5, then the prototype position of the first byte in the candidate matching data segment 1 may include at least one of the position of the first byte in the candidate matching data segment 2 or the position of the first byte in the candidate matching data segment 5. In a possible implementation, if the candidate matching data segment is the same as multiple candidate matching data segments, the candidate matching data segment directly referenced by the candidate matching data segment may be the candidate matching data segment that is closest to the candidate matching data segment.

[0121] In the symbol system provided in the embodiment of the present application, for an alternative matching data segment S[i..i+l], the device selects the alternative matched data segment directly referenced by the matching data segment S[i..i+l] as S[j..j+l], where j is less than i, that is, j is the byte before i. In the position array, set Ancestor[i+k]=Ancestor[j+k] and Ancestor[i+k]=j+k, where k is greater than or equal to 0 and k is less than l. That is, the prototype position of S[i+k] in the byte stream includes at least one of the prototype position of S[j+k] or the position of S[j+k].

[0122] In some cases, since the position of the candidate matched data segment corresponding to the candidate matching data segment can be called the left boundary of the decompressed candidate matching data segment, the position array can also be called the left boundary array. In this case, Ancestor[i] represents the left boundary of the decompressed S[i], that is, starting from the Ancestor[i]th character in S, S[i] can be decompressed.

[0123] After the position array is generated, the distance between the position of each byte and the prototype position corresponding to each byte can be determined according to the position of each byte in the position array and the prototype position corresponding to each byte, and then at least one candidate distance corresponding to each candidate matching data segment can be determined. For any candidate matching data segment, at least one candidate distance is determined based on the position of the candidate matching data segment and the position of at least one candidate matched data segment.

[0124] In a possible implementation, at least one alternative distance between each alternative matching data segment and at least one alternative matched data segment referenced by each alternative matching data segment can be determined by the position of the first byte in each alternative matching data segment and the prototype position corresponding to the first byte. For example, the alternative matched data segment directly referenced by alternative matching data segment 1 is alternative matched data segment 7, the position of the first byte in alternative matching data segment 1 in the data group is 46, and the position of the first byte in alternative matched data segment 7 in the data group is 39, then the alternative distance between alternative matching data segment 1 and alternative matched data segment 7 is 6, that is, the number of bytes between the first byte in alternative matching data segment 1 and the first byte in alternative matched data segment 7. If alternative matched data segment 7 belongs to alternative matched data segment 8, and alternative matched data segment 8 directly references alternative matched data segment 9, then alternative matched data segment 1 indirectly references alternative matched data segment 9, and the position of the byte in alternative matched data segment 9 corresponding to the first byte in alternative matched data segment 1 is 25, then the alternative distance between alternative matched data segment 1 and alternative matched data segment 9 is 20.

[0125] After determining at least one alternative distance corresponding to each candidate matching data segment, it can be determined whether each candidate matching data segment can be determined as a matching data segment based on the relative size of the alternative distance and the reference distance. For any candidate matching data segment, if at least one alternative distance corresponding to the candidate matching data segment is greater than the reference distance, it can be considered that if the candidate matching data segment is determined as a matching data segment, then when decompressing the compressed segment obtained by compressing the matching data segment, it is necessary to decompress the matching data segment based on the compressed segment corresponding to the non-matching data segment that is farther away from the compressed segment, and it is difficult to ensure the stability of decompression. Therefore, the candidate matching data segment can be determined as a non-matching data segment. In the subsequent decompression process of the non-matching data segment, the decompression can be achieved by the non-matching data segment itself, thereby improving the stability and efficiency of the decompression.

[0126] If there is an alternative distance less than or equal to the reference distance among at least one alternative distance corresponding to the alternative matching data segment, the alternative matching data segment can be determined as the matching data segment, and the alternative data segment to be matched corresponding to the alternative distance less than or equal to the reference distance can be determined as the data segment to be matched corresponding to the matching data segment. If the number of alternative distances less than or equal to the reference distance is multiple, the data segment to be matched with the smallest alternative distance can be determined as the data segment to be matched corresponding to the matching data segment.

[0127] In the symbol system provided by the embodiments of the present application, the reference distance can be identified by W. The role of the reference distance is similar to that of a sliding window, which is used to control the stable left boundary of each matching data segment. Based on the above description, it can be known that in the embodiments of the present application, the logic for finding the data segment to be matched is modified as follows: for the matching data segment S[i..i+l] to complete the matching, it is necessary that for any k (0 <= k < l), (i + k) - Ancestor[i + k] <= W. If this is not satisfied, the alternative matching data segment S[i..i+l] is re-determined as a non-matching data segment. In the embodiments of the present application, by introducing a position array and a reference distance during the compression process, it is possible to control that each matching data segment can be decompressed by a data segment to be matched whose distance from the matching data segment is within the reference distance. In some cases, after controlling the distance between the matching data segment and the data segment to be matched according to the position array, the position array can be deleted to reduce space occupancy.

[0128] In addition, when compressing data, the data processing method provided by the embodiments of the present application does not need to consider the service characteristics of the data, can adapt to various scenarios where data compression is required, and has strong scenario adaptability.

[0129] S403. Compress the multiple data segments according to the data segments referred to by the multiple data segments to obtain first compressed data including multiple compressed segments. Each compressed segment is obtained by compressing the corresponding data segment. The data segment referred to by any matching data segment is the data segment to be matched with the same bytes as those in any matching data segment, and the data segment referred to by any non-matching data segment is any non-matching data segment.

[0130] The process of compressing the multiple data segments and outputting the first compressed data including multiple compressed segments can refer to the process of compressing the multiple data segments in the general compression algorithm based on LZ77 described above, and will not be elaborated here. The embodiments of the present application do not limit the storage location of the first compressed data formed after compression. Exemplarily, after compression, the data group before compression can be deleted, and the first compressed data obtained by compression can be stored at the position originally used to store the data group, or the first compressed data can be stored at a position different from the data group. For example, Figure 5In the data processing scenario shown, the first compressed data may be stored in the storage block to which the original data group belongs, or may be independently stored in other locations in the HOT. Figure 5 ③ in the storage location indicated by ), it is necessary to ensure that any B+-Tree index established on the basis of HOT can refer to the storage location of the first compressed data. If the storage location of the first compressed data is different from the storage location of the data group, the value of RID also needs to be changed so that the changed value of RID indicates the storage location of the first compressed data. For example, Figure 5 In the process indicated by ④ in the figure, the value in R2 corresponding to data 2 in the B+-Tree changes from the value pointing to the storage location of data 2 in the data group (indicated by the dotted arrow) to the value pointing to the storage location of data 2 in the first compressed data (indicated by the solid arrow). Correspondingly, the value in R3 corresponding to data 3 in the B+-Tree changes from the value pointing to the storage location of data 3 in the data group (indicated by the dotted arrow) to the value pointing to the storage location of data 3 in the first compressed data (indicated by the solid arrow).

[0131] In order to improve the efficiency of the subsequent decompression process, the positions of multiple matching data segments and multiple non-matching data segments can be recorded during the compression process, so that the decompression can be performed quickly according to the positions of the multiple matching data segments and the positions of the multiple non-matching data segments during the decompression process. Therefore, in a possible implementation, after compressing multiple data segments according to the data segments referenced by the multiple data segments, mapping information can also be generated, and the mapping information is used to indicate the mapping relationship between the starting positions of the multiple data segments and the starting positions of the multiple compressed segments. Among them, the mapping information can also be called an anchor index (Anchor Index), and the mapping information can include the starting position of each data segment in the multiple data segments, and the starting position of each compressed segment corresponding to each data segment. Alternatively, in order to save the space occupied by the mapping information, the mapping information may not include the positions of each data segment and each compressed segment, but include the starting position of at least one reference data segment and the starting position of at least one reference compressed segment, and at least one reference data segment corresponds to at least one reference compressed segment.

[0132] Optionally, at least one reference data segment may be determined in the data group first, and then at least one reference compressed segment corresponding to the at least one reference data segment may be determined. The at least one reference data segment may be determined according to the size of the data group. For example, the first step length for selecting the reference data segment may be determined according to the size of the data group, and at least one reference data segment may be determined in the data group according to the first step length. The first step length is the distance between two adjacent reference data segments. The size of the first step length may be set based on experience or specified by the user, and the first step lengths corresponding to different data groups may be the same or different. For example, if the size of the data group is 8KB, 1KB may be used as the first step length, and the first data segment whose distance between the position before compression and the starting position of the data group is not less than 1KB, 2KB, ..., 7KB is determined respectively. The seven data segments determined are the reference data segments, and then the starting positions of the respective reference data segments may be recorded. Afterwards, it is necessary to determine the respective reference compressed segments obtained by compressing the respective reference data segments, and record the starting positions of the respective reference compressed segments in the first compressed data.

[0133] See also Figure 7 , showing a schematic diagram of mapping information. The length of the data group before compression is 18B, which is used to determine the step length of the reference data segment 6B. Then, the first data segment BCD whose distance between the position before compression and the starting position of the data group is not less than 6B can be determined as a reference data segment, and the starting position of the data segment BCD is recorded as 6 in the mapping information. Afterwards, the first data segment EA whose distance between the position before compression and the starting position of the data group is not less than 12B can be determined as another reference data segment, and the starting position of the data segment EA is recorded as 13 in the mapping information.

[0134] Taking the length of the compressed segment obtained by matching the data segment compression as 1B as an example, after determining the starting position of the reference data segment BCD, it can be determined that the reference compressed segment corresponding to the reference data segment BCD is (2, 3, 1), and the starting position of the reference compressed segment (2, 3, 1) can be recorded in the mapping information as 7. The reference compressed segment corresponding to the reference data segment EA is EA, and the distance from the starting position of the reference compressed segment EA in the mapping information can be 10.

[0135] Optionally, at least one reference compressed segment can be determined in the first compressed data first, and then at least one reference data segment corresponding to the at least one reference compressed segment can be determined. The at least one reference compressed segment can be determined according to the size of the first compressed data obtained by compression. For example, the second step length for selecting the reference compressed segment can be determined according to the size of the first compressed data, and at least one reference compressed segment can be determined in the first compressed data according to the second step length. The second step length is the distance between two adjacent reference compressed segments. The size of the second step length can be set based on experience or specified by the user. The second step lengths corresponding to first compressed data of different sizes can be the same or different. For example, if the size of the data group is 4KB, 1KB can be used as the first step length, and the first compressed segment whose distance between the compressed position and the starting position of the first compressed data is not less than 1KB, 2KB and 3KB is determined respectively. The three compressed segments determined are the reference compressed segments, and then the starting position of each reference compressed segment can be recorded. After that, it is necessary to determine the reference data segments of each reference compressed segment obtained by compression, and record the starting position of each reference data segment in the data group.

[0136] In some cases, whether the mapping information records the starting position of each data segment and each compressed segment, or the starting position of at least one reference data segment and at least one reference compressed segment, it can be represented by the symbol system provided in the embodiment of the present application. In this symbol system, the starting position of each data segment or the starting position of at least one reference data segment can be represented by AS, and the starting position of each compressed segment or the starting position of at least one reference compressed segment can be represented by AD.

[0137] In this case, the distribution characteristics of the starting position in the mapping information can be represented by an ordered binary array, that is, the mapping relationship between the starting position of the data segment and the starting position of the corresponding compressed segment data segment can be recorded by an ordered binary array.<AS,AD> or<AD,AS> .

[0138] Since the mapping information can indicate the mapping relationship between the starting positions of multiple data segments and the starting positions of multiple compressed segments, during the decompression process, the starting position of the compressed segment corresponding to the starting position of any data segment can be efficiently determined according to the mapping information, or the starting position of the data segment corresponding to the starting position of any compressed segment can be efficiently determined.

[0139] Based on the above description of the data compression method in the data processing method provided in the embodiment of the present application, it can be known that in the embodiment of the present application, after compression, not only the compressed first compressed data can be output, but also the mapping information can be output. Therefore, when storing the content output after the compression is completed, it is necessary to determine the storage format according to the format of the output content, and store the output content persistently according to the storage format. For example, the output content includes two parts, the first compressed data and the mapping information. The storage format can be a format for storing the first compressed data and the mapping information separately or a format for storing the first compressed data and the mapping information together. Different storage formats can be determined for different data processing scenarios. For example, Figure 5 In the HOT-based data processing scenario shown in FIG. 1 , the storage format may be a format in which the first compressed data and the mapping information are stored together, such as Figure 6 In the LSM-Tree-based data processing scenario shown, the storage format can be a format for storing the first compressed data and the mapping information separately. The first compressed data can be stored in SSTable, and the mapping information can be stored in MemTable.

[0140] For example, see Figure 8 , shows a schematic diagram of a data compression process, firstly obtain a data group, and determine whether the processing of all bytes is completed. If the processing of all bytes is not completed, continue to process the next byte, and determine whether a matched data segment of the byte can be found. If a matched data segment of the byte can be found, the byte is temporarily recorded as a matched data segment. If a matched data segment of the byte cannot be found, the byte is temporarily recorded as a non-matching data segment. If the processing of all bytes has been completed, the final matching data segments and non-matching data segments are determined according to the position array and the reference distance, and the first compressed data and mapping information are output.

[0141] In the present application, by controlling the distance between the matching data segment and the matched data segment in the data group to be less than or equal to the reference distance, during decompression, the decompression of each compressed segment that needs to be decompressed can be achieved based on the compressed segment corresponding to the data segment whose distance between each data segment does not exceed the reference distance. Furthermore, before decompression, the boundary of the compressed data that needs to be decompressed can be determined based on the reference distance, and the compressed data can be decompressed within the boundary, thereby ensuring stable decompression performance.

[0142] The above describes the data compression process in the data processing method provided in the embodiment of the present application. The following describes the data decompression process corresponding to the data compression process. Fig. 9 , shows a schematic diagram of a data decompression process, the data decompression process includes but is not limited to the following S901 to S903.

[0143] S901, obtain a data processing instruction, the data processing instruction instructs to decompress the target data from the first compressed data, the first compressed data is obtained by compressing the data group to which the target data belongs, and the data processing instruction includes the first starting position of the target data in the data group and the length of the target data.

[0144] The target data may be part of the data in the data group or all of the data in the data group. In the embodiment of the present application, the decompression process is described by temporarily taking the target data as part of the data in the data group.

[0145] Based on the foregoing description, it can be known that a data group can be stored in a HOT, and the data group includes multiple data in a storage block of the HOT. In this case, the target data can include at least one data among the multiple data.

[0146] In a data processing environment based on LSM-Tree, data groups can also be stored in the baseline database of LSM-Tree. In order to control the discreteness of read access and reduce the maintenance cost of incremental data changes, in a data processing environment based on LSM-Tree, block groups are used as the minimum unit of I / O writes, and storage blocks are used as the minimum unit of I / O reads. Therefore, a data group includes data in multiple storage blocks in a block group of the baseline database, and target data can include data in at least one storage block among the multiple storage blocks.

[0147] In addition, in a data processing environment based on LSM-Tree, storage blocks with a high reading frequency can be stored in a block cache. Figure 6 , Figure 6 The block cache shown in ② is used to cache storage blocks with high access frequency. In some cases, Figure 6 In the data reading process indicated by ③, data is read in units of storage blocks.

[0148] The embodiment of the present application does not limit the method for obtaining the data processing instruction. For example, when the target data needs to be obtained during the business operation, the device can autonomously generate a data processing instruction to instruct to decompress the first compressed data to obtain the target data, thereby obtaining the data processing instruction. Alternatively, the device can also receive a data processing instruction sent by other devices or input by a user, thereby obtaining the data processing instruction.

[0149] If the data processing instruction is generated by the device autonomously, the device may first obtain the first starting position of the target data in the data group and the length of the target data before generating the data processing instruction. Taking the HOT-based data processing environment as an example, the RID corresponding to the target data may be determined by searching through the B+-Tree index according to the K corresponding to the target data, and the first starting position of the target data in the data group may be determined according to the RID, and the length of the target data may be determined according to the length of at least one data included in the target data.

[0150] In the symbol system provided in the embodiment of the present application, the first starting position of the target data can be expressed as OFF, and the length of the target data can be expressed as (size, SZ).

[0151] S902, based on the first starting position and the reference distance, determine the second starting position of the second compressed data in the first compressed data, the second compressed data is used to decompress to obtain the target data, and the reference distance indicates the maximum value of the distance between any data segment in the target data and a data segment referenced by any data segment.

[0152] The embodiment of the present application does not limit the method for determining the second starting position based on the first starting position and the reference distance. In a possible implementation, the data group includes multiple data segments, the first compressed data includes multiple compressed segments, and each compressed segment is compressed by the corresponding data segment. In this implementation, the process of determining the second starting position may include: obtaining the starting position of each data segment in the data group; determining the first data segment to which the first position belongs in the multiple data segments according to the starting position of each data segment, the first position is determined based on the difference between the first starting position and the reference distance; determining the starting position of the first compressed segment obtained by compressing the first data segment according to the starting position of the first data segment; and determining the starting position of the first compressed segment as the second starting position.

[0153] The difference between the first starting position and the reference distance can indicate the first position that is before the first starting position and whose distance from the first starting position is the reference distance. Since the distance between any data segment in the target data and the data segment referenced by any data segment is less than or equal to the reference distance, that is, the data segment referenced by any data segment in the target data segment is located at the first position or after the first position, decompression is started from the starting position of the first compressed segment corresponding to the first data segment belonging to the first position, which can ensure that the target data is obtained by decompression.

[0154] In one possible implementation, obtaining the starting position of each data segment in a data group may include: obtaining the length of each data segment; determining the starting position of each data segment in the data group based on the length of each data segment and the order of each compressed segment in the first compressed data, the order of any data segment in the data group is the same as the order of the compressed segments obtained by compressing any data segment in the first compressed data.

[0155] Since the order of any data segment in the data group is the same as the order of the compressed segments obtained by compressing any data segment in the first compressed data, the order of each data segment in the data group can be determined according to the order of each compressed segment in the first compressed data, and then the initial position of each data segment in the data group can be accurately determined according to the length of each data segment and the order of each data segment in the data group.

[0156] Exemplarily, since the lengths of multiple data segments are stored in multiple compressed segments, the lengths of each data segment and the order of each compressed segment in the first compressed data can be determined by traversing each compressed segment, and the cumulative lengths of all data segments before each data segment can be determined, thereby determining the starting position of each data segment in the data group.

[0157] For example, by reading the lengths of each data segment stored in multiple compressed segments, it is determined that the length of the first data segment corresponding to the first compressed segment is 3B, then it can be determined that the starting position of the first data segment in the data group is 0, and the starting position of the second data segment corresponding to the second compressed segment in the data group is 3.

[0158] In a possible implementation, the first position for determining the first data segment among the starting positions of multiple data segments can be determined based on the difference between the first starting position and the reference distance. Exemplarily, the difference between the first starting position and the reference distance is calculated, and if the difference is a value greater than or equal to 0, the difference can be determined as the first position, and if the difference is less than 0, 0 can be determined as the first position. That is, if the difference is a value greater than or equal to 0, it means that the position indicated by the difference is a position in the data group, and if the difference is a value less than 0, it means that the difference is not a position in the data group, which can also indicate that the distance between the first starting position and the starting position of the data group is less than or equal to the reference distance, so the starting position of the data group can be determined as the first position, that is, 0 is determined as the first position.

[0159] In the symbol system provided in the embodiment of the present application, the first position can be represented as TARGET_OFF, and the relationship between the first position, the first starting position and the reference distance can be represented by TARGET_OFF=MAX(OFF-W,0), where MAX is a function for finding the maximum value.

[0160] After determining the starting position of each data segment and the first position, the first data segment to which the first position belongs can be determined from among the multiple data segments according to the starting position of each data segment, and then the starting position of the first data segment can be determined. Exemplarily, among the starting positions of each data segment, a starting position A that is closest to the first position and is smaller than or equal to the first position can be determined, and a starting position B that is closest to the first position and is larger than or equal to the first position can be determined.

[0161] If the starting position A and the starting position B are the same as the first position, it means that the first position is the starting position of the first data segment, and the data segment with the first position as the starting position can be determined as the first data segment. If the starting position A is different from the starting position B, and the starting position A is smaller than the first position, and the starting position B is larger than the first position, it means that the first position is located in the data segment with the starting position A as the starting position, and the data segment with the starting position A as the starting position can be determined as the first data segment. If the starting position A is different from the starting position B, and one of the starting positions A or B is the same as the first position, the data segment corresponding to the starting position that is the same as the first position can be determined as the first data segment.

[0162] In a possible implementation, the starting position of the first compressed segment obtained by compressing the first data segment is determined according to the starting position of the first data segment, including: obtaining mapping information, the mapping information is used to indicate the mapping relationship between the starting positions of multiple data segments and the starting positions of multiple compressed segments; determining the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment. The mapping information is the mapping information generated during the compression process. Since the mapping information may include the starting position of each data segment and the starting position of each compressed segment, and may also include the starting position of at least one reference data segment and the starting position of at least one reference compressed segment, the determination of the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment can be divided into different situations according to the different contents included in the mapping information, for example, it can be divided into the following situations A1 and A2.

[0163] In case A1, the mapping information includes the starting position of each data segment and the starting position of each compressed segment. In case A1, in the process of determining the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment, the starting position of the first compressed segment having a mapping relationship with the first data segment can be directly found through the starting position of the first data segment. For example, in some cases, the mapping information stores a pair corresponding to the starting position of each data segment and the starting position of each compressed segment, and then the pair to which the starting position AS1 of the first data segment belongs can be found in each pair.<AS1,AD1> or binary<AD1,AS1> , AD1 in the tuple is the starting position of the first compressed segment, and the starting position of the first compressed segment is the second starting position of the second compressed data.

[0164] Case A2, the mapping information includes the starting position of at least one reference data segment in each data segment and the starting position of at least one reference compressed segment obtained by compressing the at least one reference data segment. In case A2, in the process of determining the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment, a first reference data segment is determined in at least one reference data segment according to the starting position of the first data segment and the starting position of at least one reference data segment, and the first reference data segment is a reference data segment that is before the first data segment and is closest to the first data segment; the starting position of the first reference compressed segment corresponding to the first reference data segment is determined as the starting position of the first compressed segment.

[0165] Taking the binary group corresponding to the starting position of at least one reference data segment and the starting position of at least one reference compressed segment stored in the mapping information as an example, the first reference data segment closest to the starting position AS1 of the first data segment and after the first data segment can be found according to the starting position of at least one reference data segment in each binary group, the starting position of the first reference data segment is AS2, and the binary group to which AS2 belongs is used to find the first reference data segment that is closest to the starting position AS1 of the first data segment and after the first data segment.<AS2,AD2> or binary<AD2,AS2> The starting position AD2 of the first reference compression segment is determined. The starting position AD2 of the first reference compression segment is the starting position of the first compression segment, and the starting position of the first compression segment is the second starting position of the second compressed data.

[0166] S903, decompressing the second compressed data according to the second starting position and the ending position of the second compressed data to obtain target data, wherein the ending position of the second compressed data is determined based on the first starting position and the length of the target data.

[0167] In a possible implementation, before decompressing the second compressed data according to the second starting position and the ending position of the second compressed data to obtain the target data, it is also necessary to determine the ending position of the second compressed data. The embodiment of the present application does not limit the method for determining the ending position of the second compressed data. For example, the second data segment to which the second position belongs can be determined in multiple data segments according to the starting position of each data segment, and the second position is the sum of the length of the first starting position and the target data; the starting position of the third data segment adjacent to the second data segment is determined in the starting position of each data segment, and the third data segment is after the second data segment; according to the starting position of the third data segment, the starting position of the second compressed segment corresponding to the third data segment is determined; the starting position of the second compressed segment is determined as the ending position of the second compressed data.

[0168] The sum of the first starting position and the length of the target data can indicate the end position of the target data in the data group, that is, the second position can indicate the end position of the target data in the data group, and the second data segment to which the second position belongs is the data segment to which the end position of the target data belongs. Therefore, the starting position of the third data segment adjacent to the second data segment and after the second data segment is located after the end position of the target data, and decompression is stopped from the starting position of the second compressed segment corresponding to the third data segment, which can ensure that the target data is completely decompressed.

[0169] In one possible implementation, the method for determining the second data segment to which the second position belongs among multiple data segments according to the starting positions of each data segment can refer to the method for determining the first data segment to which the first position belongs among multiple data segments according to the starting positions of each data segment in S902, which will not be repeated here.

[0170] After determining the second data segment, the third data segment after the second data segment and adjacent to the second data segment can be determined, and the starting position of the third data segment can be determined in the starting positions of each data segment, and then the starting position of the second compressed segment corresponding to the third data segment can be determined according to the starting position of the third data segment. The second compressed segment corresponding to the third data segment can be a compressed segment obtained by compressing the third data segment, or it can be a reference compressed segment corresponding to a reference data segment that is closest to the third data segment and after the third data segment. The method for determining the starting position of the second compressed segment according to the starting position of the third data segment can refer to the method for determining the starting position of the first compressed segment according to the starting position of the first data segment in S902, which will not be repeated here. The determined starting position of the second compressed segment is the end position of the second compressed data.

[0171] After determining the second starting position and the second compressed data end position, the second compressed data can be decompressed according to the second starting position and the second compressed data end position to obtain the target data. In some cases, the data range of the second compressed data is larger than the data range of the target data, that is, the data obtained by decompressing the second compressed data includes the target data and data other than the target data. In this case, decompressing the second compressed data according to the second starting position and the second compressed data end position to obtain the target data includes: decompressing the second compressed data according to the second starting position and the second compressed data end position to obtain intermediate data; determining the target data end position according to the first starting position and the length of the target data; and intercepting the target data in the intermediate data according to the first starting position and the target data end position.

[0172] Optionally, the process of decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain the intermediate data can refer to the description of the general data decompression process based on LZ77 in the previous text, which will not be repeated here. In addition, in a possible implementation, the second compressed data can also be decompressed starting from the second starting position, and after completing the decompression of each compressed segment, the end position of the data segment obtained by decompressing each compressed segment is recorded, and when the end position of a recorded data segment is greater than or equal to the second position, the decompression is stopped to obtain the intermediate data.

[0173] The decompressed intermediate data includes the target data, so the target data can be intercepted from the intermediate data. The starting position of the target data is the first starting position, so before intercepting the target data, the end position of the target data needs to be determined. Exemplarily, the sum of the first starting position and the length of the target data can be determined as the end position of the target data. For example, the first starting position is 9, the length of the target data is 56B, and the end position of the target data can be determined to be 64. Then, the data between the first starting position 9 and the end position 64 can be intercepted from the first data to obtain the target data.

[0174] In a possible implementation, the intermediate data includes a compressed segment that is not referenced by any data segment in the target data, and the compressed segment referenced by the compressed segment is located outside the intermediate data, that is, the compressed segment referenced by the compressed segment is located before the second starting position, resulting in the compressed segment being unable to be decompressed and restored. However, since each data segment in the target data does not reference the compressed segment that cannot be decompressed and restored, even if there is a compressed segment that cannot be decompressed and restored in the intermediate data, it does not affect the acquisition of the target data.

[0175] In addition, when the present application needs to query or call part of the data (such as a single-point query on a data segment or field), there is no need to decompress all the compressed data, thereby improving the decompression efficiency, reducing resource waste, and avoiding affecting the operating efficiency of the business that needs to call or query the data.

[0176] In summary, since the reference distance indicates the maximum value of the distance between any data segment in the target data and the data segment referenced by any data segment, the data segment referenced by each data segment in the target data may be located in the target data, or in the partial data whose distance from the first starting position is less than or equal to the reference distance. Then, according to the first starting position and the reference distance, the second starting position of the second compressed data used for decompressing the target data can be determined, and it is ensured that the target data can be obtained by decompressing the second compressed data from the second starting position. In addition, according to the first starting position and the length of the target data, the end position of the second compressed data can be accurately determined, and the decompression is stopped at the end position of the second compressed data, which not only ensures that the decompressed data includes the target data, but also avoids the decompression of compressed data that is not necessary to be decompressed, thereby improving the decompression efficiency and reducing resource waste.

[0177] The present application uses the determined second starting position and the end position of the second compressed data as the decompression boundary, so that the decompression performance according to the clear decompression boundary is stable.

[0178] The data processing method of the embodiment of the present application is introduced above. Corresponding to the above method, the embodiment of the present application also provides a data processing device. Fig.10 or Fig.11 is a schematic diagram of the structure of the data processing device provided in the embodiment of the present application. The device can implement the aforementioned data processing method through software, hardware, or a combination of both, and the device can be applied to a device that provides cloud computing services. It should be understood that the device may include more additional modules than the modules shown or omit some of the modules shown therein, and the embodiment of the present application does not limit this.

[0179] like Fig.10 The data processing device shown comprises:

[0180] An acquisition module 1001 is used to acquire a data processing instruction, the data processing instruction indicates decompressing target data from first compressed data, the first compressed data is obtained by compressing a data group to which the target data belongs, and the data processing instruction includes a first starting position of the target data in the data group and a length of the target data; a determination module 1002 is used to determine a second starting position of second compressed data in the first compressed data based on the first starting position and a reference distance, the second compressed data is used to decompress and obtain the target data, the reference distance indicates the maximum value of the distance between any data segment in the target data and a data segment referenced by any data segment; a decompression module 1003 is used to decompress the second compressed data according to the second starting position and an end position of the second compressed data to obtain the target data, the end position of the second compressed data is determined based on the first starting position and the length of the target data.

[0181] In a possible implementation, the first compressed data includes multiple compressed segments, the data group includes multiple data segments, and each compressed segment is compressed by the corresponding data segment; a determination module 1002 is used to obtain the starting position of each data segment in the data group; according to the starting position of each data segment, determine the first data segment to which the first position belongs in the multiple data segments, and the first position is determined based on the difference between the first starting position and a reference distance; according to the starting position of the first data segment, determine the starting position of the first compressed segment corresponding to the first data segment; and determine the starting position of the first compressed segment as the second starting position.

[0182] In one possible implementation, the determination module 1002 is used to obtain the length of each data segment; based on the length of each data segment and the order of each compressed segment in the first compressed data, the starting position of each data segment in the data group is determined, and the order of any data segment in the data group is the same as the order of the compressed segment obtained by compressing any data segment in the first compressed data.

[0183] In one possible implementation, the determination module 1002 is used to obtain mapping information, where the mapping information is used to indicate a mapping relationship between the starting positions of multiple data segments and the starting positions of multiple compressed segments; and determine the starting position of the first compressed segment based on the mapping information and the starting position of the first data segment.

[0184] In one possible implementation, the mapping information includes the starting position of at least one reference data segment in the multiple data segments and the starting position of at least one reference compressed segment compressed by the at least one reference data segment; the determination module 1002 is used to determine the first reference data segment in the at least one reference data segment according to the starting position of the first data segment and the starting position of the at least one reference data segment, the first reference data segment being the reference data segment that is before the first data segment and is closest to the first data segment; the starting position of the first reference compressed segment obtained by compressing the first reference data segment is determined as the starting position of the first compressed segment.

[0185] In a possible implementation, the determination module 1002 is further used to determine, in multiple data segments, the second data segment to which the second position belongs, based on the starting positions of each data segment, where the second position is the sum of the first starting position and the length of the target data; determine, in the starting positions of each data segment, the starting position of a third data segment adjacent to the second data segment, where the third data segment is after the second data segment; determine, based on the starting position of the third data segment, the starting position of the second compressed segment corresponding to the third data segment; and determine the starting position of the second compressed segment as the end position of the second compressed data.

[0186] In one possible implementation, the decompression module 1003 is used to decompress the second compressed data according to the second starting position and the ending position of the second compressed data to obtain intermediate data; determine the ending position of the target data according to the first starting position and the length of the target data; and intercept the target data in the intermediate data according to the first starting position and the ending position of the target data.

[0187] In a possible implementation, the data group is stored in the HOT, the data group includes multiple data in a storage block of the HOT, and the target data includes at least one data among the multiple data.

[0188] In a possible implementation, the data group is stored in a baseline database of the LSM-Tree, the data group includes data in multiple storage blocks in the baseline database, and the target data includes data in at least one storage block among the multiple storage blocks.

[0189] like Fig.11 The data processing device shown comprises:

[0190] An acquisition module 1101 is used to acquire a data group, where the data group includes multiple bytes; a division module 1102 is used to divide the multiple bytes into multiple data segments, where the multiple data segments include at least one of a matching data segment or a non-matching data segment, where the bytes in the non-matching data segment are different from the bytes in each data segment before the non-matching data segment, where the bytes in the matching data segment are the same as the bytes in a matched data segment before the matching data segment, and where the distance between the matching data segment and the matched data segment is less than or equal to a reference distance; a compression module 1103 is used to compress the multiple data segments based on the data segments referenced by the multiple data segments to obtain first compressed data including multiple compressed segments, where each compressed segment is compressed by the corresponding data segment, where the data segment referenced by any matching data segment is the matched data segment that has the same bytes as those in any matching data segment, and where the data segment referenced by any non-matching data segment is any non-matching data segment.

[0191] In one possible implementation, the division module 1102 is used to divide multiple bytes into multiple alternative data segments, the multiple alternative data segments include at least one of alternative matching data segments or non-matching data segments, and the bytes in the alternative matching data segment are the same as the bytes in at least one alternative matched data segment before the alternative matching data segment; obtain a position array of the alternative matching data segment, the position array includes the position of the alternative matching data segment and the position of at least one alternative matched data segment corresponding to the alternative matching data segment; when there is an alternative distance less than or equal to the reference distance in at least one alternative distance, determine the alternative matching data segment as a matching data segment, and at least one alternative distance is determined based on the position of the alternative matching data segment and the position of at least one alternative matched data segment; when at least one alternative distance is greater than the reference distance, determine the alternative matching data segment as a non-matching data segment.

[0192] In a possible implementation, the device further includes a generation module; the generation module is used to generate mapping information, where the mapping information is used to indicate a mapping relationship between the starting positions of the multiple data segments and the starting positions of the multiple compressed segments.

[0193] In a possible implementation, a data group is stored in a HOT, and the data group includes a plurality of data in a storage block of the HOT.

[0194] In a possible implementation, the data group is stored in a baseline database of the LSM-Tree, and the data group includes data in multiple storage blocks in the baseline database.

[0195] It should be understood that the above Fig.10 or Fig.11When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0196] The present application also provides a computing device that can be configured as a server in the above implementation environment. Fig.12 , Fig.12 Schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. Fig.12 As shown, computing device 600 includes: bus 602, processor 604, memory 606 and communication interface 608. Processor 604, memory 606 and communication interface 608 communicate through bus 602. It should be understood that the present application does not limit the number of processors and memories in computing device 600.

[0197] The bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 The bus 602 may include a path for transmitting information between various components of the computing device 600 (eg, the memory 606, the processor 604, and the communication interface 608).

[0198] The processor 604 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0199] The memory 606 may include a volatile memory, such as a random access memory (RAM). The processor 604 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0200] The memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the aforementioned modules, thereby implementing the data processing method. That is, the memory 606 stores instructions for executing the data processing method.

[0201] The communication interface 608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or communication networks.

[0202] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be configured as a server in the above implementation environment, such as a central server, an edge server, or a local server in a local data center.

[0203] Fig.13 Schematic diagram of a computing device cluster provided in an embodiment of the present application. Fig.13 As shown, the computing device cluster includes at least one computing device 600. The memory 606 in one or more computing devices 600 in the computing device cluster may store the same instructions for executing the data processing method.

[0204] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 600 may jointly execute instructions for executing the data processing method.

[0205] It should be noted that the memory 606 in different computing devices 600 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data processing apparatus. That is, the instructions stored in the memory 606 in different computing devices 600 can implement the functions of one or more modules in each module.

[0206] In some embodiments, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. Fig.14 Schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application. Fig.14 As shown, two computing devices 600 are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0207] It should be understood that Fig.14 The functionality of the computing device 600 shown in FIG. 6 may also be performed by multiple computing devices 600 .

[0208] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0209] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0210] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0211] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element.

[0212] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0213] The term "at least one" in this application means one or more, and the term "multiple" in this application means two or more, for example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0214] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0215] It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0216] It should also be understood that the terms "if" and "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0217] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0218] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0219] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0220] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the data involved in this application are all obtained with full authorization.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data processing instruction, the data processing instruction instructing to decompress target data from first compressed data, the first compressed data being obtained by compressing a data group to which the target data belongs, the data processing instruction including a first starting position of the target data in the data group and a length of the target data; Obtaining the starting position of each data segment in the data group; Determine, according to the starting positions of the respective data segments, a first data segment to which a first position belongs, wherein the first position is determined based on a difference between the first starting position and a reference distance, wherein the reference distance indicates a maximum value of a distance between any data segment in the target data and a data segment referenced by any data segment, wherein the data segment referenced by any data segment includes a non-matching data segment or a matched data segment having the same bytes as those in any data segment; Determining, according to the starting position of the first data segment, a starting position of a first compressed segment obtained by compressing the first data segment; Determine the starting position of the first compression segment as the second starting position of second compressed data in the first compressed data, the second compressed data being used for decompressing to obtain the target data; The second compressed data is decompressed according to the second starting position and the ending position of the second compressed data to obtain the target data, wherein the ending position of the second compressed data is determined based on the first starting position and the length of the target data.

2. The method according to claim 1, characterized in that: The obtaining the starting position of each data segment in the data group includes: Obtaining the length of each data segment; The starting position of each data segment in the data group is determined according to the length of each data segment and the order of each compressed segment obtained by compressing the data segment in the first compressed data, and the order of any data segment in the data group is the same as the order of the compressed segment obtained by compressing any data segment in the first compressed data.

3. The method according to claim 1 or 2, characterized in that: The determining, according to the starting position of the first data segment, the starting position of the first compressed segment obtained by compressing the first data segment comprises: Acquire mapping information, where the mapping information is used to indicate a mapping relationship between starting positions of a plurality of data segments in the data group and starting positions of a plurality of compressed segments in the first compressed data; The starting position of the first compressed segment is determined according to the mapping information and the starting position of the first data segment.

4. The method according to claim 3, characterized in that The mapping information includes a starting position of at least one reference data segment among the multiple data segments and a starting position of at least one reference compressed segment obtained by compressing the at least one reference data segment; determining the starting position of the first compressed segment according to the mapping information and the starting position of the first data segment includes: Determine a first reference data segment in the at least one reference data segment according to the starting position of the first data segment and the starting position of the at least one reference data segment, wherein the first reference data segment is a reference data segment that is before the first data segment and is closest to the first data segment; A starting position of a first reference compressed segment obtained by compressing the first reference data segment is determined as a starting position of the first compressed segment.

5. The method according to claim 1, 2 or 4, characterized in that: Before decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain the target data, the method further includes: Determine, according to the starting positions of the respective data segments, a second data segment to which a second position belongs among the plurality of data segments in the data group, wherein the second position is the sum of the first starting position and the length of the target data; Determine a starting position of a third data segment adjacent to the second data segment among the starting positions of the data segments, wherein the third data segment is after the second data segment; Determine, according to the starting position of the third data segment, the starting position of the second compressed segment corresponding to the third data segment; The starting position of the second compression segment is determined as the ending position of the second compressed data.

6. The method according to claim 3, characterized in that: Before decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain the target data, the method further includes: Determine, according to the starting positions of the respective data segments, a second data segment to which a second position belongs among the plurality of data segments in the data group, wherein the second position is the sum of the first starting position and the length of the target data; Determine a starting position of a third data segment adjacent to the second data segment among the starting positions of the data segments, wherein the third data segment is after the second data segment; Determine, according to the starting position of the third data segment, the starting position of the second compressed segment corresponding to the third data segment; The starting position of the second compression segment is determined as the ending position of the second compressed data.

7. The method according to claim 1, 2, 4 or 6, characterized in that: The decompressing the second compressed data according to the second starting position and the ending position of the second compressed data to obtain the target data includes: decompressing the second compressed data according to the second starting position and the end position of the second compressed data to obtain intermediate data; Determining an end position of the target data according to the first starting position and the length of the target data; The target data is intercepted from the intermediate data according to the first starting position and the end position of the target data.

8. The method according to claim 1, 2, 4 or 6, characterized in that: The data group is stored in a heap organization table HOT, the data group includes a plurality of data in a storage block of the HOT, and the target data includes at least one data among the plurality of data.

9. The method according to claim 1, 2, 4 or 6, characterized in that: The data group is stored in a baseline database of a log structure merge tree, the data group includes data in a plurality of storage blocks in the baseline database, and the target data includes data in at least one storage block of the plurality of storage blocks.

10. A data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire a data processing instruction, wherein the data processing instruction instructs to decompress target data from first compressed data, wherein the first compressed data is obtained by compressing a data group to which the target data belongs, and the data processing instruction includes a first starting position of the target data in the data group and a length of the target data; A determination module, used to obtain the starting position of each data segment in the data group; The determination module is further configured to determine, according to the starting positions of the respective data segments, a first data segment to which a first position belongs, wherein the first position is determined based on a difference between the first starting position and a reference distance, wherein the reference distance indicates a maximum value of a distance between any data segment in the target data and a data segment referenced by any data segment, wherein the data segment referenced by any data segment includes a non-matching data segment or a matched data segment having the same bytes as those in any data segment; The determining module is further used to determine, according to the starting position of the first data segment, a starting position of a first compressed segment obtained by compressing the first data segment; The determining module is further used to determine the starting position of the first compressed segment as the second starting position of the second compressed data in the first compressed data, and the second compressed data is used to decompress to obtain the target data; A decompression module is used to decompress the second compressed data according to the second starting position and the end position of the second compressed data to obtain the target data, and the end position of the second compressed data is determined based on the first starting position and the length of the target data.

11. The device according to claim 10, characterized in that The determination module is used to obtain the length of each data segment; determine the starting position of each data segment in the data group according to the length of each data segment and the order of each compressed segment obtained by compressing the each data segment in the first compressed data, and the order of any data segment in the data group is the same as the order of the compressed segment obtained by compressing any data segment in the first compressed data.

12. The device according to claim 10 or 11, characterized in that The determination module is used to obtain mapping information, where the mapping information is used to indicate a mapping relationship between the starting positions of multiple data segments in the data group and the starting positions of multiple compressed segments in the first compressed data; and determine the starting position of the first compressed segment based on the mapping information and the starting position of the first data segment.

13. The device according to claim 12, characterized in that The mapping information includes the starting position of at least one reference data segment among the multiple data segments and the starting position of at least one reference compressed segment compressed by the at least one reference data segment; the determination module is used to determine a first reference data segment in the at least one reference data segment according to the starting position of the first data segment and the starting position of the at least one reference data segment, the first reference data segment being a reference data segment that is before the first data segment and is closest to the first data segment; and determine the starting position of the first reference compressed segment obtained by compressing the first reference data segment as the starting position of the first compressed segment.

14. The device according to claim 10, 11 or 13, characterized in that The determination module is further used to determine, according to the starting positions of the respective data segments, a second data segment to which a second position belongs among the multiple data segments in the data group, wherein the second position is the sum of the first starting position and the length of the target data; determine, among the starting positions of the respective data segments, a starting position of a third data segment adjacent to the second data segment, wherein the third data segment is after the second data segment; determine, according to the starting position of the third data segment, a starting position of a second compressed segment corresponding to the third data segment; and determine the starting position of the second compressed segment as the ending position of the second compressed data.

15. The device according to claim 12, characterized in that The determination module is further used to determine, according to the starting positions of the respective data segments, a second data segment to which a second position belongs among the multiple data segments in the data group, wherein the second position is the sum of the first starting position and the length of the target data; determine, among the starting positions of the respective data segments, a starting position of a third data segment adjacent to the second data segment, wherein the third data segment is after the second data segment; determine, according to the starting position of the third data segment, a starting position of a second compressed segment corresponding to the third data segment; and determine the starting position of the second compressed segment as the ending position of the second compressed data.

16. The device according to claim 10, 11, 13 or 15, characterized in that The decompression module is used to decompress the second compressed data according to the second starting position and the end position of the second compressed data to obtain intermediate data; determine the end position of the target data according to the first starting position and the length of the target data; and intercept the target data in the intermediate data according to the first starting position and the end position of the target data.

17. The device according to claim 10, 11, 13 or 15, characterized in that The data group is stored in a heap organization table HOT, the data group includes a plurality of data in a storage block of the HOT, and the target data includes at least one data among the plurality of data.

18. The device according to claim 10, 11, 13 or 15, characterized in that The data group is stored in a baseline database of a log structure merge tree, the data group includes data in a plurality of storage blocks in the baseline database, and the target data includes data in at least one storage block of the plurality of storage blocks.

19. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device includes a processor, and the processor is coupled to a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method according to any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1 to 9.

Images