A data block and parallel reading storage and query method thereof

Abstract

The present invention discloses a data block and a storage and query method for parallel reading thereof. The data block includes: a data block header, a compressed data area, a compression buffer, an original data area, and a compression directory table, which are divided in order from beginning to end. The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data segments and whether there is uncompressed data in the original data area. The compression directory table is used to store the starting position, length, and compression type of each compressed data segment in the compressed data area. In the data block, the original data will not be overwritten before the compressed data can be read. In the data block, all modifications to the written data are performed in the form of atomic operations. The present invention has the advantages of small memory usage, high memory utilization, no waiting for reading and writing, no "phantom read" situation, and lock-free parallel reading while writing.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and more specifically to a data block storage and query method for parallel reads thereof; in particular, to a data block storage and query method that supports parallel reads of the same data block during compression. The present invention can be widely applied to storage and query services in a variety of fields, including industrial databases, vector databases for large language models, time series databases, and internet search. Background Art

[0002] A data block is a unit of memory or disk used to store data, typically 4KB or multiples thereof. Multiple data values ​​generated continuously at specific intervals are typically stored in the same data block. Because most data values ​​have a reasonable range and their changing trends are predictable, they can be compressed to save storage and access overhead. Raw data is typically appended to a data block individually or in batches. Once a data block is full, there are two common approaches to handling it:

[0003] (1) Compress the data block content and write the compressed smaller data block to the disk. However, this operation will cause inconsistent data block sizes on the disk, affecting data block access performance;

[0004] (2) Compress the data block content and continue to write the original data into the free space generated after compression, and repeat this process.

[0005] All of the aforementioned methods have a common problem: during compression, since the data format in the data block changes, compression of a data block is usually an exclusive operation, and other read operations are not allowed at the same time, otherwise dirty data is easily read.

[0006] To address the problem of being unable to support parallel reads of the same block during data compression, there are currently five traditional solutions:

[0007] 1) The compressed data block copy is stored in another location and can exchange access permissions with the original data block. However, this method theoretically takes up twice as much storage space;

[0008] 2) Place a cache before the data block, close to the user, to store data values ​​that may be compressed at any time. This method is currently the most commonly used, but due to the cache expiration policy and the fact that the data block is completely full and does not need to be cached, the synchronization time overhead is very high without affecting performance, so the actual storage space wasted is far more than doubled.

[0009] 3) After a data block is compressed once, no new data is accepted and the reduced data block is written directly to the disk. This method results in inconsistent sizes written to the disk, which affects block access performance.

[0010] 4) Hide the data being compressed and then make it visible after compression is complete. However, this can cause the query result data size to fluctuate, which is known as "phantom reads" in the industry.

[0011] 5) During data compression, the compression buffer is located outside the data block and is locked only during the process of replacing the original data within the block with the compressed data outside the block. This replacement process essentially rewrites all or part of the data block from the original data format to the compressed data format. Even if locked during the update process, the data format state cannot be atomically switched, making lock-free reading and writing impossible. Ultimately, all read and write operations must be locked, significantly reducing concurrency performance.

[0012] Therefore, those skilled in the art urgently need to solve the problem of excessive memory usage and reduced concurrency performance caused by the inability to provide read services simultaneously during the data compression process within a data block in the existing technology. Summary of the Invention

[0013] In view of this, the present invention provides a data block and parallel reading storage and query method thereof, which can solve the problem in the prior art of excessive memory usage and reduced concurrency performance caused by the inability to provide read services simultaneously during data compression within the data block.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] In a first aspect, an embodiment of the present invention provides a data block, comprising: a data block header, a compressed data area, a compression buffer area, an original data area, and a compression directory table, which are respectively divided in order from beginning to end;

[0016] The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data segments and whether there is uncompressed data in the original data area;

[0017] The compressed data area is used to store compressed data of all data segments;

[0018] The compression buffer is configured to: trigger compression when the original data area is full or reaches a preset threshold; the compressed data is stored in the compression buffer;

[0019] The original data area is configured such that each data value written into the data block by the user is initially concentrated in the area to wait for compression;

[0020] The compression directory table is used to store the starting position, length and compression type of each compressed data segment in the compressed data area;

[0021] In a data block, the original data will not be overwritten before the compressed data can be read; and in a data block, all modifications to the written data are performed atomically.

[0022] Preferably, a data area to be allocated is provided between the data block header and the compressed data area;

[0023] and / or a data area to be allocated is provided between the compression buffer area and the original data area;

[0024] And / or a data area to be allocated is provided between the original data area and the compressed directory.

[0025] Preferably, the compression buffer area is the same size as the original data area.

[0026] Preferably, the initial value of the integer variable is 0;

[0027] When the first data in the data block is written into the original data area, the initial value is atomically incremented by 1 to become an odd value;

[0028] When the data in the original data area is compressed and provided for access in the compressed data area, the odd value is atomically increased by 1 to become an even number.

[0029] Preferably, the compressed directory table grows from the end of the block to the beginning of the block.

[0030] Preferably, the number of entries in the compression directory table is one more than the number of actual compressed data segments in the data block; the extra entry is used to reserve space for a newly added compression directory entry.

[0031] Preferably, the storage order of the entries in the compressed directory table is in reverse order.

[0032] Preferably, the original data area is further configured to record the number of data items or bytes currently written into the original data area, and store them at the end of the original data area.

[0033] In a second aspect, an embodiment of the present invention further provides a data block parallel reading storage and query method, using any data block as described in the first aspect, when writing data, the method includes:

[0034] Append the original data written by the user to the original data area of ​​the data block until it is full;

[0035] Compressing the original data in the original data area and storing the compressed data in a compression buffer in the data block;

[0036] Allocate the compressed data in the compression buffer to the compressed data area to become a new compressed data fragment;

[0037] Add a new compressed directory table entry in the compressed directory table to record the information of the new compressed data segment; make the newly generated compressed data area visible to the user;

[0038] The position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; and the data value to be written is appended to the original data area.

[0039] Furthermore, the position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; including:

[0040] Between the compressed data area and the compressed directory table, a compression buffer area and an original data area are sequentially arranged;

[0041] The compression buffer and the original data area have the same capacity; or the compression buffer and the original data area are divided according to the previous compression ratio of the original data; or the compression buffer and the original data area are divided according to the prediction result of the next compression ratio output by the pre-trained model.

[0042] In a third aspect, an embodiment of the present invention further provides a storage and query method for parallel reading of data blocks, using any data block as described in the first aspect. When reading data, the method includes:

[0043] Get the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments;

[0044] Obtain the position, size and compression type of each compressed data fragment in the compressed data area;

[0045] Read the compressed data from the compressed data area, decompress it in the external buffer and read the original data;

[0046] Determine whether the last data fragment of the current data block is in the original data area;

[0047] If so, record the transaction point where the data has been read, and then read the data from the original data area;

[0048] After reading the original data, recheck whether the data after the transaction point is valid;

[0049] If the verification fails, the data after the transaction point needs to be reread from the compressed data area.

[0050] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following advantages:

[0051] The compression process eliminates the need for any buffers outside of the data block. The compressed data generated after compression requires no copying or movement. From the start of compression until the compressed data fragment is ready for service, all written data can be read concurrently and without locks. This approach offers advantages such as a small memory footprint, high memory utilization, zero wait times for both reads and writes, no "phantom reads," and lock-free parallel reads while writing, making it suitable for high-concurrency applications in data-related service environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 This is a schematic diagram of the functional area division of the data block provided by the present invention.

[0054] Figure 2 This is a schematic diagram of the principle of reading compressed segments and storing compressed directory tables provided by the present invention.

[0055] Figure 3 This is a schematic diagram of the storage principle of the original data area provided by the present invention.

[0056] Figure 4 This is a schematic diagram of the data block writing principle provided by the present invention.

[0057] Figure 5 The data block writing data flow chart provided by the present invention.

[0058] Figure 6 This is a schematic diagram of the data block reading principle provided by the present invention.

[0059] Figure 7 The data block reading data flow chart provided by the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The present invention provides a data block and a storage and query method for parallel reading thereof. To clearly describe the technical solution of the present invention, the following unified explanations of the technical terms involved in the text are provided as follows:

[0062]

[0063]

[0064] Example 1:

[0065] The embodiment of the present invention discloses a data block, referring to Figure 1 As shown, it includes: a data block header, a compressed data area, a compressed buffer area, an original data area and a compressed directory table which are divided in order from the beginning to the end;

[0066] The data block header has an atomically modifiable integer variable that is used to store two states: the number of compressed data fragments and whether there is uncompressed data in the original data area.

[0067] The compressed data area follows the data block header and is used to store the compressed data of all data segments. The data segments are stored closely together without any gaps. The compressed data area grows towards the end of the block.

[0068] The compression buffer is configured so that compression is triggered when the original data area is full or reaches a preset threshold; the compressed data is stored in the compression buffer. The starting position of the compression buffer moves toward the end of the block as the compressed data area grows. The compression buffer is located immediately after the compressed data area within the block, without any gaps. This ensures that compressed data is not copied, improving performance.

[0069] The original data area is configured so that each data value written by the user into the data block is initially gathered in this area and awaits compression.

[0070] The compression directory table is used to store the starting position, length and compression type of each compressed data segment in the compressed data area;

[0071] Within a data block, the original data will not be overwritten before the compressed data can be read, without exception. Furthermore, within a data block, all modifications to already written data are performed atomically, without exception. Furthermore, within a data block, except for the compressed data area and the compression buffer, which must be adjacent and seamless, more data areas can be inserted before and after other areas to facilitate subsequent functional expansion. For example, there is a to-be-allocated data area between the data block header and the compressed data area; another example is a to-be-allocated data area between the compression buffer and the original data area; and another example is a to-be-allocated data area between the original data area and the compression directory.

[0072] The unallocated data area is a specific area within a data block that has not yet been used or allocated for specific data. This area can be used to store future data or functionality, allowing expansion or the addition of new functionality without reorganizing existing data. This design provides flexibility, allowing these reserved areas to be adjusted and allocated as needed to accommodate new data, metadata, or other information. This allows for easy functional expansion without impacting existing data or functionality.

[0073] The data block provided by the present invention is designed with a core mechanism that ensures that data can be read in parallel without locks at the same time as it is written, which helps to realize storage and query that supports parallel reading of the same data block during compression. In addition, based on the partitioning of the data block, there is no need to rely on any buffer outside the data block during the compression process, and all is completed within the data block. The compressed data generated after compression does not require any copying or moving, and is directly written to the compression buffer. The location of the compression buffer is the final location of the data fragment in the data block and does not need to be moved. It helps to achieve "from the start of compression to the time when the compressed data fragment can provide services, all written data can be provided with lock-free read access in parallel to the outside world." In addition, the functional area division of the data block has the advantages of small memory usage, high memory utilization, no waiting for reading and writing, no "phantom reading", and lock-free parallel reading while writing. It is suitable for use in high-concurrency application environments for data-related services.

[0074] In one embodiment, the data block header is a small area at the beginning of the data block used to store certain specific configurations. The data block header contains an atomically modifiable integer variable that stores two states: the number of compressed data fragments and whether there is uncompressed data in the original data area. This variable is 0 when the block is empty; when the first piece of data in the data block is written to the original data area, the value is atomically incremented by 1 to become an odd number; and when the data in the original data area is compressed and accessible in the compressed data area, the value is atomically incremented by 1 to become an even number.

[0075] The essence of this mechanism is a finite state machine, which can represent multiple states through a linearly changing value while ensuring the order of states observed from a temporal perspective. This invention uses the example of recording two states in this variable as an example. As the functionality is expanded in the future, more ordered states can be represented in this state machine, implementing more complex storage logic.

[0076] In one embodiment, referring to Figure 2As shown, the compression directory table stores the starting position, length, and compression type of each compressed data segment in the compressed data area. If the compressed data is larger than the original data, the original data is stored. The compression directory table grows from the end of the block toward the beginning of the block. The number of entries in the compression directory table exceeds the number of compressed data segments in the data block by one. This extra entry reserves space for new compression directory entries. Entries in the compression directory table are stored in reverse order; as the number of compressed data segments increases, the compression directory table grows toward the beginning of the block.

[0077] Reference Figure 2 As shown, the directory table entry corresponding to the first compressed data fragment is stored at the first position from the end of the data block to the block header, the second compressed data fragment is stored at the second position from the end of the data block to the block header, and so on.

[0078] Calculation of the total length of the compressed directory table:

[0079] [Compressed directory table entry length * (compressed data fragment number + 1)];

[0080] The position of the compressed directory table entry corresponding to the data fragment in the original data area in the data block is calculated as follows: [data block length - compressed directory table entry length * (number of compressed data fragments + 1)].

[0081] For example, [Compression Directory Table Entry 1] stores the starting position, length, and compression type of compressed data segment 1; [Compression Directory Table Entry 2] stores the starting position, length, and compression type of compressed data segment 2. The extra entry is used to reserve space for the newly added compression directory entry.

[0082] In one embodiment, referring to Figure 3 As shown in the figure, the original data area is located after the compression buffer and is specifically used to store the original data values ​​written by the user but not yet compressed. The compression buffer is the same size as the original data area.

[0083] Considering the need for parallel reading, the original data area also records the number of data entries or bytes currently written to the original data area and stores it at the very end of the original data area. This value is only used to obtain the length of the original data during the parallel reading process, but does not participate in compression. When the original data area is full or reaches the preset threshold, compression is triggered; one or more conditions may be met, including but not limited to:

[0084] (1) The remaining space in the original data area can no longer accommodate more data values; (2) The original data area has reached a certain quantity or capacity limit (for example: 1000 entries, 512 bytes, etc.).

[0085] The size of the compressed buffer is the same as that of the original data area. The calculation method of the original data area position is:

[0086] [Data block header length + compressed data area length + compressed buffer length];

[0087] Calculation method of original data area size:

[0088] [(data block length - data block header length - compressed data area length - compressed directory table length) / 2].

[0089] Example 2:

[0090] Based on the data block of Example 1, the present invention also provides a parallel reading storage and query method, as step number StepA: Figure 4-5 As shown, when writing data, it includes:

[0091] (1) First, obtain the location and capacity of the original data area;

[0092] (2) The data written by the user is then appended to the original data area until the original data area is full or reaches a certain threshold and cannot accommodate the data value to be written;

[0093] (3) then compressing the original data in the original data area within the block and storing the compressed data in the compression buffer;

[0094] (4) The compressed data in the compression buffer is then allocated to the end of the compressed data area to form a new compressed data segment, without the compressed data being moved. This step is only a logical step, and the physical placement of the actual compressed data in the compressed data area is already ready.

[0095] (5) Then, a new compressed directory table entry is added to the compressed directory table to record the information of the compressed fragment just generated;

[0096] (6) Modify the data block header so that the compressed data segment just generated is visible to the user while the original data area is invisible to the user;

[0097] (7) Then, the position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block, and the data value to be written is appended to the original data area.

[0098] Step B: Refer to Figure 6-7 As shown, when reading data:

[0099] 1) First, read the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments;

[0100] 2) Then obtain the position, size and compression type of each compressed data segment in the compressed data area from the compression directory table entry;

[0101] 3) Then read the compressed data from the compressed data area, decompress it through the data block external buffer, and read the original data;

[0102] 4) Then determine whether the data block has the last data fragment stored in the original data area;

[0103] 5) If the last data fragment is in the original data area, the current state of the read data must be recorded as a transaction point for easy backtracking;

[0104] 6) Then directly read the original data from the original data area;

[0105] 7) After all the original data has been read, it is determined whether the original data area where the original data segment just read is located has a compression state change;

[0106] 8) If the check finds that the data in the original data area has been compressed, the data read from the original data area needs to be backtracked and the data segment needs to be read again from the compressed data area.

[0107] In this embodiment, based on the data blocks of embodiment 1, lock-free parallel reading can be achieved while data is being written. When writing data, except for the area between the compressed data area and the compressed buffer area, areas can be inserted before and after other areas to store more types of data.

[0108] The number of compressed data segments and whether there is uncompressed data in the original data area are stored in an integer variable in the data block header (hereinafter referred to as

[0109] compress_piece_count_score). This value is an integer variable modified by atomic operations. When the block is empty, the value is 0; when the first data is written to the original data area, the value is atomically incremented by 1 to become an odd number; when the data in the original data area is compressed and can be accessed in the compressed data area, the value is atomically incremented by 1 to become an even number. When appending data to the data block,

[0110] The storage mechanism of compress_piece_count_score, as the last step of the compression process, increases the number of compressed data pieces by 1 through an atomic operation, and converts the original data area from available to unavailable.

[0111] The following are detailed descriptions of Step A and Step B:

[0112] 1. Step A writes data

[0113] In the above step A, the external data written by the user can be fixed-length, variable-length, or compressed according to the characteristics of the original data. The present invention does not care about the content and format of the original data, nor does it require that each data value must be fixed-length.

[0114] In the above step Step A, considering the need for parallel reading, after writing the original data to the original data area, the length of the data written to the current original data area (hereinafter referred to as orignal_bytes) must be recorded in the original data area, or the number of bytes must be recorded and stored at the end of the original data area.

[0115] In Step A, the compressed data generated from the original data is stored in the compression buffer. Since the original data area and the compression buffer are the same size, the compressed data length should not be larger than the original data. Otherwise, the original data is directly copied to the compression buffer, and the compression method of the corresponding compression directory entry marks this data segment as uncompressed.

[0116] At this time, since the compress_piece_count_score value is not atomically increased by 1 to become an even number, the modification of the content of the area reserved for the newly added compressed fragment in the compression directory table will not be read by the parallel read logic.

[0117] In the above step Step A: the calculation method for obtaining the number of compressed data pieces in the data block is: if the compress_piece_count_score value is an even number, then the number of compressed data pieces is

[0118] [compress_piece_count_score / 2]; if the compress_piece_count_score value is an odd number, the number of compressed data fragments is [(compress_piece_count_score-1) / 2].

[0119] The calculation method for determining whether there is uncompressed data in the original data area of ​​the data block is: if the compress_piece_count_score value is an even number, it means that there is no uncompressed data in the original data area; if the compress_piece_count_score value is an odd number, it means that there is still uncompressed data in the original data area.

[0120] In Step A above, the action of making the newly generated compressed data piece visible to the user while simultaneously hiding the original data area from the user is an atomic operation. Before the compression operation is complete, the value of compress_piece_count_score is an odd number. The final step of the compression operation atomically increments the value of compress_piece_count_score by 1, increasing the number of newly generated compressed data pieces by 1 and making compress_piece_count_score an even number, thus completing the process of hiding the original data area from the user.

[0121] Furthermore, if compression is successful, the compress_piece_count_score value is atomically incremented by 1 to become an even number, and the original data area is now unreadable. The original_bytes value, which records the length of data currently written to the original data area, is cleared to 0, completing the action of clearing the original data area. In addition, because part of the compression buffer was previously allocated to the compressed data area, after clearing the original data area, the size of the compression buffer and the original data area must be recalculated in the remaining space. The calculation method is as follows:

[0122] First: If the size of the compressed buffer is the same as that of the original data area, the calculation method of the original data area position is:

[0123] [Data block header length + compressed data area length + compressed buffer length];

[0124] Calculation method of original data area size:

[0125] [(data block length - data block header length - compressed data area length - compressed directory table length) / 2].

[0126] Second: Divide the compression buffer and the original data area according to the previous compression ratio of the original data; or divide the compression buffer and the original data area according to the prediction result of the next compression ratio output by the pre-trained model.

[0127] For example, if a user knows the compression ratio of their own data, or can estimate the compression ratio of the next data based on the actual compression ratio of the previous data block. Optionally, for example, generative AI can be used in a bypass manner to predict the compression ratio of the next batch of data based on the actual compression ratio of the previously compressed data segment and the data distribution trend. In the process of dividing the size of the compression buffer and the original data area, the area size can be divided according to the previously calculated ratio. For example, if the compression ratio of the next compressed data segment is predicted to be above 1:10, the ratio of the compression buffer to the original data area can be configured based on 1:10, which can greatly improve data compression efficiency.

[0128] In Step A, the write process is stateless. The caller can stop appending data after any successful write to the block. At this point, the data block can be written directly to disk, the network, or other media for subsequent reading. If the data block's own data validation mechanism is not considered, the algorithm does not need to perform any finishing work at this point.

[0129] 2. Step B reads data

[0130] In the above step B, when reading data, the number of compression directory entries in the data block is calculated using the compress_piece_count_score in the data block header. That is, if the compress_piece_count_score value is an even number, the number of compressed data fragments is:

[0131] N = [compress_piece_count_score / 2]. Since the number of entries in the compression directory table is 1 more than the number of compressed data pieces in the data block, N+1 is the final number of compression directory entries. However, only each entry corresponding to all compressed data pieces, i.e., N entries, is traversed.

[0132] If the compress_piece_count_score value is odd, the number of compressed data pieces is

[0133] M = [(compress_piece_count_score - 1) / 2]. Since the number of entries in the compression directory table is 1 more than the number of compressed data pieces in the data block, M + 1 is the final number of compression directory entries. However, only the entries corresponding to all compressed data pieces, i.e., M entries, are traversed.

[0134] In Step B, the compressed data segments are decompressed using an external buffer. A data block can only be written by one thread at a time, but can be read concurrently by multiple threads or processes. Therefore, using an external buffer for decompression ensures parallel reads and facilitates data caching.

[0135] The external buffer is used for decompression during data reading. To support maximum concurrency, a buffer size of no more than one data block can be set for each CPU. Regardless of the concurrency and user connections supported by the entire system, only the number of decompression buffers required for reading needs to be equal to the number of CPU cores. This buffer can be implemented using thread-local storage provided by the operating system.

[0136] In Step B above, are there any uncompressed data fragments in the current data block? Which data fragment is the last compressed fragment in the current block? Is the last data fragment stored in the original data area and not yet compressed? The answer to these three questions can be obtained by looking at the value of a variable called compress_piece_count_score in the data block header:

[0137] Answer to Question 1: The calculation method is based on the parity of the compressed_piece_count_score value; an odd number indicates that there is an uncompressed data segment; an even number indicates that there is no uncompressed data segment.

[0138] Answer to Question 2: The calculation method is based on the even value of compress_piece_count_score; compress_piece_count_score / 2 indicates the number and the last compressed piece.

[0139] Answer to Question 3: The calculation method is to use the odd value of compress_piece_count_score to determine whether the last data fragment is stored in the original data area and has not been compressed; and (compress_piece_count_score+1) / 2 indicates the number and the last uncompressed data fragment.

[0140] In Step B, when reading data from a data block currently being written, if dirty data is encountered, a backtracking reread is required. Dirty data can only be encountered while reading the original data area. Therefore, before reading data from the original data area, the status of the currently read data must be recorded as a transaction point (hereinafter referred to as transaction_begin). This allows the original data just read to be deleted and reread if an inconsistency in the read status is detected.

[0141] In step B, before determining that data needs to be read from the original data area, the compress_piece_count_score value in the current block header must be saved. Firstly, after the data reading is completed, the value must be read again and the difference must be compared to determine whether the status of the original data area has changed. Secondly, in case of subsequent data conflicts, the position of the corresponding compressed data area needs to be calculated when the compressed data area is reread.

[0142] Furthermore, after reading from the original data area, you must first

[0143] The parity of the compress_piece_count_score value determines whether there are uncompressed data segments. The current compress_piece_count_score value is then compared with the compress_piece_count_score value before the data was read from the original data area. If the compress_piece_count_score values ​​in the two read data block headers differ, this indicates that during the original data read, the original data in the compression buffer was compressed and written to the compression buffer. After the compression buffer was allocated to the compressed data area, the starting position of the original data area has changed or data belonging to another data segment has been written. The data previously read from the original data area needs to be discarded. Backtracking the previously recorded transaction_begin can speed up the discarding process.

[0144] The "start position of the original data area has changed" refers to the situation where, during the data reading process, the original data area is compressed due to writing, and then the original data area is recalculated, causing the starting position of the original data area to move toward the end of the block. At the beginning of the data reading, the original data area is closer to the block header than its current position, which will inevitably cause data errors.

[0145] Furthermore, the subscript of the compressed data area in the original data area to be read is calculated as follows: (compress_piece_count_score value + 1) / 2. The recorded compress_piece_count_score value must be an odd number.

[0146] That is, before reading the original data, the compress_piece_count_score value must be an odd number, indicating that there is data in the compressed data area. After reading the original data area, judge again

[0147] If the compress_piece_count_score value is not equal, the compressed data area must be read again. The compressed_piece_count_score value saved before reading the original data + 1 divided by 2 is the index of the compression directory table entry. Locating the compression directory table entry can naturally locate the specific location and size of the compressed data area.

[0148] Therefore, if verification fails after reading data from the original data area, the data must be in the compressed data area. After deleting the data already read from the original data area, the compressed data is reread from the compressed data area. This process only needs to be repeated once, without the need for repeated reading. That is, after reading the contents of the original data area, if a conflict with a parallel write is found, resulting in an inconsistent state, the corresponding compressed area data can be reread without locking. The backtracking operation only needs to be performed once, without repeated checks until the state is consistent. Therefore, it can be concluded that the possibility of backtracking due to a conflict when reading the original data is very small. And because the value of compress_piece_count_score is updated atomically, each backtracking and rereading action not only does not require locking, but also only needs to be performed once.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0150] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data block, characterized in that: include: The data block header, compressed data area, compressed buffer area, original data area and compressed directory table are divided in order from the beginning to the end; The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data segments and whether there is uncompressed data in the original data area. The initial value of the integer variable is 0. When the first data in the data block is written to the original data area, the initial value is atomically incremented by 1 to become an odd value. When the data in the original data area is compressed and provided for access in the compressed data area, the odd value is atomically incremented by 1 to become an even number. The compressed data area is used to store the compressed data of all data segments; the data segments are stored closely without any gaps; The compression buffer is configured to: trigger compression when the original data area is full or reaches a preset threshold; the compressed data is stored in the compression buffer; the compressed data area and the compression buffer are adjacent to each other without a gap; The original data area is configured such that each data value written into the data block by the user is initially concentrated in the area to wait for compression; The compression directory table is used to store the starting position, length, and compression type of each compressed data segment in the compressed data area; if the compressed data is larger than the original data, the original data is stored; the compression directory table grows from the end of the block to the beginning of the block; the number of entries in the compression directory table is one more than the number of actual compressed data segments in the data block; the extra entry is used to reserve space for the newly added compression directory entry; the storage order of the entries in the compression directory table is in reverse order; In a data block, the original data will not be overwritten before the compressed data can be read; and in a data block, all modifications to the written data are performed atomically. The compression buffer and the original data area have the same capacity; or the compression buffer and the original data area are divided according to the previous compression ratio of the original data; When writing data, the original data written by the user is appended to the original data area of ​​the data block until the original data area is full or reaches a preset threshold; the compression operation is triggered, the original data in the original data area is compressed and stored in the compression buffer; the compressed data in the compression buffer is allocated to the compressed data area to form a new compressed data fragment; a new compression directory table entry is added to the compression directory table to record the information of the new compressed data fragment; the newly generated compressed data area is made visible to the user; the position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; and the data value to be written is appended to the original data area; When reading data, obtain the number of entries in the compression directory table in the data block; traverse each entry corresponding to all compressed data fragments; obtain the position, size and compression type of each compressed data fragment in the compressed data area; read the compressed data from the compressed data area, decompress it in the external buffer, and read the original data; determine whether the last data fragment of the current data block is in the original data area; if so, record the transaction point where the data has been read, and then read the data from the original data area; after reading the original data, re-check whether the data after the transaction point is valid; if the check fails, the data after the transaction point needs to be re-read from the compressed data area.

2. A data block according to claim 1, characterized in that: A data area to be allocated is provided between the data block header and the compressed data area; and / or a data area to be allocated is provided between the compression buffer area and the original data area; And / or a data area to be allocated is provided between the original data area and the compressed directory.

3. The data block according to claim 1, wherein: The original data area is further configured to record the number of data items or bytes currently written into the original data area and store the data at the end of the original data area.

4. A data block parallel reading storage and query method, characterized in that: Using the data block according to any one of claims 1 to 3, when writing data, the method comprises: Append the original data written by the user to the original data area of ​​the data block until it is full; Compressing the original data in the original data area and storing the compressed data in a compression buffer in the data block; Allocate the compressed data in the compression buffer to the compressed data area to become a new compressed data fragment; Add a new compressed directory table entry in the compressed directory table to record the information of the new compressed data segment; make the newly generated compressed data area visible to the user; Recalculate the position and capacity of the original data area and the compression buffer in the remaining space of the data block; Append the data value to be written to the original data area.

5. A data block parallel reading storage and query method, characterized in that: Using the data block according to any one of claims 1 to 3, when reading data, the method comprises: Get the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments; Obtain the position, size and compression type of each compressed data fragment in the compressed data area; Read the compressed data from the compressed data area, decompress it in the external buffer and read the original data; Determine whether the last data fragment of the current data block is in the original data area; If so, record the transaction point where the data has been read, and then read the data from the original data area; After reading the original data, recheck whether the data after the transaction point is valid; If the verification fails, the data after the transaction point needs to be reread from the compressed data area.

Images