Data block and parallel read storage and query method thereof
By implementing atomic operations and lock-free reading and writing in the data block, the problem of inability to read in parallel during data compression in the data block is solved, concurrency performance and memory utilization are improved, and it is suitable for high-concurrency application environments.
Patent Information
- Application Number
- CN202411110991.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-14
AI Technical Summary
In the prior art, read services cannot be provided simultaneously during data compression in data blocks, resulting in excessive memory usage and degradation of concurrency performance.
Adopt a data block structure, including data block header, compressed data area, compressed buffer, original data area and compressed directory table. Through atomic operation and lock-free reading and writing mechanism, compression and reading are completed in the data block to avoid dependence on external Buffer to ensure lock-free parallel read access during compression.
It realizes that there is no need to wait for read and write operations during the compression process, with small memory usage and high utilization rate, avoiding the phenomenon of "illusion reading" and is suitable for high concurrent application environments.
Smart Images

Figure CN119025523A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and more specifically to a data block and a storage and query method for parallel reading thereof; in particular, to a data block and a storage and query method for supporting parallel reading of the same data block during compression. The present invention can be widely used in storage and query services in multiple fields such as industrial databases, vector databases of large language models, time series databases, and Internet searches. Background Art
[0002] A data block is a memory or disk unit used to store data, and is generally 4K or multiples thereof. Multiple data values that are continuously generated at a specific interval are usually stored in the same data block. Since most data values have a reasonable range of values and their changing trends are predictable, they can be compressed to save storage and access costs. Raw data is usually appended to a data block one by one or in batches. When a data block is full, there are usually two ways to process it:
[0003] (1) Compress the data block content and write the compressed smaller data block to the disk. However, this operation will cause inconsistent data block sizes on the disk, affecting data block access performance.
[0004] (2) Compress the data block content and continue to write the original data into the free space generated after compression, and repeat this process.
[0005] No matter which of the above methods is used, there is a common problem: during compression, since the data format in the data block changes, compression of a data block is usually an exclusive operation, and other read operations are not allowed at the same time, otherwise dirty data is easily read.
[0006] To solve the problem of not being able to support parallel reading of the same block during data compression, there are currently five traditional methods:
[0007] 1) The compressed data block copy is stored in another place, and the access rights can be exchanged with the original data block. However, this method theoretically takes up twice the storage space;
[0008] 2) Place a cache near the user before the data block to store data values that may be compressed at any time. This method is currently the most commonly used, but because of the cache expiration policy and the fact that the data block is completely full and does not need to be cached, the synchronization time overhead is very large without affecting performance, so the actual storage space wasted is far more than doubled;
[0009] 3) After the data block is compressed once, it no longer accepts new data and writes the reduced data block directly to the disk. This method causes the size written to the disk to be inconsistent, which affects the block access performance.
[0010] 4) Set the data fragment being compressed to be hidden, and then make it visible after the compression is completed. However, this will cause the amount of query result data to fluctuate, which is called "phantom read" in the industry;
[0011] 5) When compressing data, the compression buffer is outside the data block and is only locked when the compressed data outside the data block replaces the original data inside the data block. The essence of this replacement process is to rewrite all or part of the content of the data block from the original data format to the compressed data format. Even if the update process is locked, the data format state cannot be switched atomically, so lock-free reading and writing cannot be implemented, resulting in all read and write operations having to be locked, which significantly reduces concurrency performance.
[0012] Therefore, the technical personnel in this field need to solve the problem of excessive memory usage and reduced concurrency performance caused by the inability to provide read services simultaneously during the data compression process in the data block in the prior art. Summary of the invention
[0013] In view of this, the present invention provides a data block and parallel reading storage and query method thereof, which can solve the problem of excessive memory usage and reduced concurrency performance caused by the inability to provide read services simultaneously during data compression in the data block in the prior art.
[0014] In order to achieve the above object, the present invention adopts the following technical solution:
[0015] In a first aspect, an embodiment of the present invention provides a data block, including: a data block header, a compressed data area, a compressed buffer area, an original data area, and a compressed directory table, which are respectively divided in order from the beginning to the end;
[0016] The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data fragments and whether there is uncompressed data in the original data area;
[0017] The compressed data area is used to store compressed data of all data fragments;
[0018] The compression buffer is configured to: trigger compression when the original data area is full or reaches a preset threshold; the compressed data is stored in the compression buffer;
[0019] The original data area is configured such that each data value written into the data block by the user is initially concentrated in the area to wait for compression;
[0020] The compression directory table is used to store the starting position, length and compression type of each compressed data segment in the compressed data area;
[0021] In a data block, the original data will not be overwritten before the compressed data can be read; and in a data block, all modifications to the written data are in the form of atomic operations.
[0022] Preferably, a data area to be allocated is provided between the data block header and the compressed data area;
[0023] And / or a data area to be allocated is provided between the compression buffer and the original data area;
[0024] And / or a data area to be allocated is provided between the original data area and the compressed directory.
[0025] Preferably, the compression buffer area is the same size as the original data area.
[0026] Preferably, the initial value of the integer variable is 0;
[0027] When the first data in the data block is written into the original data area, the initial value is atomically increased by 1 to become an odd value;
[0028] When the data in the original data area is compressed and provided for access in the compressed data area, the odd value is atomically increased by 1 to become an even number.
[0029] Preferably, the compression directory table grows from the end of the block to the head of the block.
[0030] Preferably, the number of entries in the compression directory table is one more than the number of actually compressed data segments in the data block; the extra entry is used to reserve space for a newly added compression directory entry.
[0031] Preferably, the storage order of the entries in the compression directory table is in reverse order.
[0032] Preferably, the original data area is further configured to: record the number of data items or bytes currently written into the original data area, and store them at the end of the original data area.
[0033] In a second aspect, an embodiment of the present invention further provides a storage and query method for parallel reading of data blocks, using a data block as described in any one of the first aspects, when writing data, the method includes:
[0034] Append the original data written by the user to the original data area of the data block until it is full;
[0035] Compressing the original data in the original data area and storing the compressed data in a compression buffer in the data block;
[0036] Allocate the compressed data in the compression buffer to the compressed data area to become a new compressed data fragment;
[0037] Add a new compressed directory table entry in the compressed directory table to record the information of the new compressed data segment; make the newly generated compressed data area visible to the user;
[0038] The position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; and the data value to be written is appended to the original data area.
[0039] Further, the position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; including:
[0040] Between the compressed data area and the compressed directory table, a compression buffer area and an original data area are sequentially arranged;
[0041] The compression buffer and the original data area have the same capacity; or the compression buffer and the original data area are divided according to the previous compression ratio of the original data; or the compression buffer and the original data area are divided according to the prediction result of the next compression ratio output by the pre-trained model.
[0042] In a third aspect, an embodiment of the present invention further provides a storage and query method for parallel reading of data blocks, using a data block as described in any one of the first aspects, when reading data, the method includes:
[0043] Get the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments;
[0044] Obtain the position, size and compression type of each compressed data fragment in the compressed data area;
[0045] Read the compressed data from the compressed data area, and read the original data after decompression in the external buffer;
[0046] Determine whether the last data fragment of the current data block is in the original data area;
[0047] If so, record the transaction point where the data has been read, and then read the data from the original data area;
[0048] After reading the original data, recheck whether the data after the transaction point is valid;
[0049] If the verification fails, the data after the transaction point needs to be re-read from the compressed data area.
[0050] It can be seen from the above technical solution that compared with the prior art, the present invention has the following advantages:
[0051] It does not need to rely on any buffer other than the data block during the compression process. The compressed data generated after compression does not need any copying or moving. From the beginning of compression to the process of compressed data fragments being able to provide services, all written data can be provided with lock-free read access in parallel. It has the advantages of small memory usage, high memory utilization, no waiting for reading and writing, no "phantom reading", and lock-free parallel reading while writing. It is suitable for use in high-concurrency application environments of data-related services. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0053] Figure 1 This is a schematic diagram of the functional area division of the data block provided by the present invention.
[0054] Figure 2 A schematic diagram of the principle of reading compressed segments and storing a compressed directory table provided by the present invention.
[0055] Figure 3 A schematic diagram of the storage principle of the original data area provided by the present invention.
[0056] Figure 4 This is a schematic diagram of the data block writing data provided by the present invention.
[0057] Figure 5 The data block provided by the present invention is written into the data flow chart.
[0058] Figure 6 This is a schematic diagram of the data block reading data provided by the present invention.
[0059] Figure 7 The data block reading data flow chart provided by the present invention. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0061] The present invention provides a data block and a storage and query method for parallel reading thereof. In order to clearly describe the technical solution of the present invention, the technical terms involved in the text are uniformly explained as follows:
[0062]
[0063]
[0064] Embodiment 1:
[0065] The embodiment of the present invention discloses a data block, referring to Figure 1 As shown, it includes: a data block header, a compressed data area, a compressed buffer area, an original data area and a compressed directory table which are respectively divided in order from the beginning to the end;
[0066] The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data fragments and whether there is uncompressed data in the original data area;
[0067] The compressed data area is located after the data block header and is used to store the compressed data of all data fragments. The data fragments are stored closely without any gaps. The compressed data area grows toward the end of the block.
[0068] The compression buffer is configured as follows: when the original data area is full or reaches the preset threshold, compression is triggered; the compressed data is stored in the compression buffer. The starting position of the compression buffer moves towards the end of the block as the compressed data area grows. The compression buffer is stored closely after the compressed data area in the block, without any interval, and the two must be adjacent and without gaps; this ensures that the compressed data is not copied and improves performance.
[0069] The original data area is configured so that each data value written by the user into the data block is initially gathered in this area and awaits compression.
[0070] The compression directory table is used to store the starting position, length and compression type of each compressed data segment in the compressed data area;
[0071] In a data block, before the compressed data can be read, the original data will not be overwritten, without exception; and in a data block, all modifications to the written data are in the form of atomic operations, without exception. In addition, in a data block, except that the compressed data area and the compressed buffer must be adjacent to each other without gaps, more data areas can be inserted before and after other areas for subsequent function expansion. For example, there is a data area to be allocated between the data block header and the compressed data area; another example is a data area to be allocated between the compressed buffer and the original data area; another example is a data area to be allocated between the original data area and the compressed directory.
[0072] The unallocated data area refers to a specific area reserved in the data block that has not yet been used or allocated by specific data. These areas can be used to store data or functions that need to be added in the future, allowing expansion or adding new functions without reorganizing existing data. This design provides flexibility, and these reserved areas can be adjusted and allocated as needed to accommodate new data, metadata, or other information. It allows easy functional expansion without affecting existing data and functions.
[0073] The data block provided by the present invention is designed with a core mechanism to ensure that data can be read in parallel without locks while being written, which helps to realize storage and query that supports parallel reading of the same data block during compression. And based on the partitioning of the data block, there is no need to rely on any buffer outside the data block during the compression process, and all are completed inside the data block. The compressed data generated after compression does not require any copying or movement, and is directly written into the compression buffer, and the location of the compression buffer is the final location of the data fragment in the data block, and does not need to be moved. It helps to achieve "from the beginning of compression to the process in which the compressed data fragment can provide services, all written data can be provided with lock-free read access in parallel". In addition, the functional area division of the data block has the advantages of small memory usage, high memory utilization, no waiting for reading and writing, no "phantom reading", and lock-free parallel reading while writing, etc., which is suitable for use in high-concurrency application environments of data-related services.
[0074] In one embodiment, the data block header is a small area located at the head of the data block, which is used to store some specific configurations. There is an atomically modifiable integer variable in the data block header, which stores two states: the number of compressed data fragments and whether there is uncompressed data in the original data area. The value is 0 when the block is empty; when the first data in the data block is written to the original data area, the value is atomically increased by 1 to become an odd number; when the data in the original data area is compressed and can be accessed in the compressed data area, the value is atomically increased by 1 to become an even number.
[0075] The essence of this mechanism is a finite state machine, which can represent multiple states through a linearly changing value while ensuring the orderliness of the states observed from a time perspective. The present invention is described by taking two states recorded in the variable as an example. With the expansion of functions in the future, more ordered states can also be represented in the state machine to achieve more complex storage logic.
[0076] In one embodiment, referring to Figure 2As shown in the figure, the compression directory table is used to store the starting position, length and compression type of each compressed data fragment in the compressed data area. If the compressed data is larger than the original data, the original data is stored. The compression directory table grows from the end of the block to the head of the block. The number of entries in the compression directory table is one more than the number of actual compressed data fragments in the data block. Among them, the extra entry is used to reserve space for the newly added compression directory table entry. The storage order of the entries in the compression directory table is in reverse order; the compression directory table grows toward the head of the block as the number of compressed data fragments increases.
[0077] Reference Figure 2 As shown, the directory table entry corresponding to the first compressed data fragment is stored in the first position from the end of the data block to the block header, the second compressed data fragment is stored in the second position from the end of the data block to the block header, and so on.
[0078] Calculation of the total length of the compressed directory table:
[0079] [Compressed directory entry length * (compressed data fragment number + 1)];
[0080] The position calculation of the compressed directory table entry corresponding to the data fragment in the original data area in the data block is: [data block length-compressed directory table entry length*(number of compressed data fragments+1)].
[0081] For example, [Compression Directory Table Item 1] stores the starting position, length, and compression type of compressed data segment 1; [Compression Directory Table Item 2] stores the starting position, length, and compression type of compressed data segment 2. The extra table item is used to reserve space for the newly added compression directory table item.
[0082] In one embodiment, referring to Figure 3 As shown in FIG. 1 , the original data area is located after the compression buffer area and is specifically used to store the original data values written by the user but not yet compressed. The size of the compression buffer area is the same as that of the original data area.
[0083] Considering the need for parallel reading, the original data area also records the number of data items or bytes currently written into the original data area and stores them at the end of the original data area. This value is only used to obtain the length of the original data during the parallel reading process, but does not participate in compression. When the original data area is full or reaches the preset threshold, compression is triggered; one or more conditions may be met, including but not limited to:
[0084] (1) The remaining space in the original data area can no longer accommodate more data values; (2) The original data area has reached a certain quantity or capacity limit (for example: 1000 records, 512 bytes, etc.).
[0085] Among them, the size of the compression buffer is the same as that of the original data area, and the calculation method of the position of the original data area is:
[0086] [Data block header length + compressed data area length + compressed buffer length];
[0087] Calculation method of the original data area size:
[0088] [(data block length - data block header length - compressed data area length - compressed directory table length) / 2].
[0089] Embodiment 2:
[0090] Based on the data block of Example 1, the present invention also provides a parallel read storage and query method, as step number StepA: Figure 4-5 As shown, when writing data, it includes:
[0091] (1) First, obtain the location and capacity of the original data area;
[0092] (2) Then the data written by the user is appended to the original data area until the original data area is full or reaches a certain threshold and cannot accommodate the data value to be written;
[0093] (3) then compressing the original data in the original data area within the block, and storing the compressed data in the compression buffer;
[0094] (4) Then the compressed data in the compression buffer is allocated to the end of the compressed data area to become a new compressed data fragment, and the compressed data does not need to be moved. This step is only a logical step, and the physical placement of the actual compressed data in the compressed data area is ready.
[0095] (5) Then, a new compression directory table entry is added to the compression directory table to record the information of the compressed fragment just generated;
[0096] (6) Modify the data block header so that the compressed data segment just generated is visible to the user while the original data area is invisible to the user;
[0097] (7) Then, the position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block, and the data value to be written is appended to the original data area.
[0098] Step B: Refer to Figure 6-7 As shown, when reading data:
[0099] 1) First, read the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments;
[0100] 2) Then obtain the position, size and compression type of each compressed data segment in the compressed data area from the compression directory table entry;
[0101] 3) Then read the compressed data from the compressed data area, decompress it through the data block external buffer, and read the original data;
[0102] 4) Then determine whether the data block has the last data fragment stored in the original data area;
[0103] 5) If the last data fragment is in the original data area, the current state of the read data needs to be recorded as a transaction point for easy backtracking;
[0104] 6) Then directly read the original data from the original data area;
[0105] 7) After all the original data are read, it is determined whether the original data area where the original data segment just read is located has a compression state change;
[0106] 8) If the current check finds that the data in the original data area has been compressed, the data read from the original data area needs to be backtracked and the data fragment needs to be read again from the compressed data area.
[0107] In this embodiment, based on the data block of embodiment 1, lock-free parallel reading can be achieved while data is being written. When data is written, except that content cannot be inserted between the compressed data area and the compressed buffer area, areas can be inserted before and after other areas to store more types of data.
[0108] The number of compressed data segments and whether there is uncompressed data in the original data area are stored in an integer variable in the data block header (hereinafter referred to as
[0109] compress_piece_count_score). This value is an integer variable modified by atomic operations. When the block is empty, the value is 0; when the first data is written to the original data area, the value is atomically increased by 1 to become an odd number; when the data in the original data area is compressed and accessible in the compressed data area, the value is atomically increased by 1 to become an even number. In the process of appending data to the data block,
[0110] The storage mechanism of compress_piece_count_score, as the last step of the compression process, increases the number of compressed data pieces by 1 through an atomic operation, and converts the original data area from available to unavailable.
[0111] The following are detailed descriptions of the above steps StepA and StepB:
[0112] 1. Step A writes data
[0113] In the above step Step A, the external data written by the user can be fixed-length, variable-length, or data compressed according to the characteristics of the original data. The present invention does not care about the content and format of the original data, nor does it require that each data value must be fixed-length.
[0114] In the above step StepA, considering the need for parallel reading, after writing the original data to the original data area, the length of the data written to the current original data area (hereinafter referred to as orignal_bytes) must be recorded in the original data area, or the number of bytes must be recorded and stored at the end of the original data area.
[0115] In the above step Step A, the compressed data generated from the original data is stored in the compression buffer. Since the size of the original data area is the same as that of the compression buffer, the length of the compressed data should not be greater than the original data, otherwise the original data is directly copied to the compression buffer, and the data segment is marked as uncompressed in the compression method of the corresponding compression directory entry.
[0116] At this time, since the compress_piece_count_score value is not atomically increased by 1 to become an even number, the modification of the content of the area reserved for the newly added compressed fragment in the compression directory table will not be read by the parallel read logic.
[0117] In the above step StepA: the calculation method for obtaining the number of compressed data pieces in the data block is: if the compress_piece_count_score value is an even number, then the number of compressed data pieces is
[0118] [compress_piece_count_score / 2]; if the compress_piece_count_score value is an odd number, the number of compressed data fragments is [(compress_piece_count_score-1) / 2].
[0119] The calculation method for determining whether there is uncompressed data in the original data area of the data block is: if the compress_piece_count_score value is an even number, it means that there is no uncompressed data in the original data area; if the compress_piece_count_score value is an odd number, it means that there is still uncompressed data in the original data area.
[0120] In the above step StepA, the action of "making the compressed data piece just generated visible to the user while the original data area is invisible to the user" is an atomic operation. Before the compression action is completed, the value of compress_piece_count_score is an odd number. The last step of the compression action is to atomically increase the value of compress_piece_count_score by 1, so that the number of compressed data pieces just generated is increased by 1, and compress_piece_count_score also becomes an even number, completing the action of making the original data area invisible to the user.
[0121] Furthermore, if the compression is successful, the compress_piece_count_score value is atomically increased by 1 to become an even number, and the original data area is unreadable. The original_bytes value that records the length of the data written to the current original data area mentioned above is cleared to 0, and the action of clearing the original data area is completed. In addition, because part of the compression buffer was previously allocated to the compression data area, after clearing the original data area, it is necessary to recalculate the size of the compression buffer and the original data area in the remaining space. The calculation method is as follows:
[0122] First: If the size of the compressed buffer is the same as that of the original data area, the calculation method of the original data area position is:
[0123] [Data block header length + compressed data area length + compressed buffer length];
[0124] Calculation method of the original data area size:
[0125] [(data block length - data block header length - compressed data area length - compressed directory table length) / 2].
[0126] Second: divide the compression buffer and the original data area according to the previous compression ratio of the original data; or divide the compression buffer and the original data area according to the prediction result of the next compression ratio output by the pre-trained model.
[0127] For example, if the user knows the compression ratio of his own data, or can estimate the compression ratio of the next data based on the actual compression ratio of the previous data block. Optionally, for example, through generative AI in a bypass manner, based on the actual compression ratio of the previous compressed data fragment and the data distribution trend, the compression ratio of the next batch of data is predicted. In the process of dividing the size of the compression buffer and the original data area, the area size can be divided according to the previously calculated ratio. For example, if it is predicted that the compression ratio of the next compressed data fragment can reach 1:10 or more, the ratio of the compression buffer to the original data area can be configured based on 1:10, which can greatly improve the data compression efficiency.
[0128] In step A, the writing process is stateless. The caller can stop appending data after any successful writing of data to the block. At this time, the data block can be directly written to the disk, network or other media for subsequent reading. If the data verification mechanism of the data block itself is not considered, the algorithm does not need to do any finishing work at this time.
[0129] 2. Step B reads data
[0130] In the above step Step B, when reading data, the number of compressed directory entries in the data block is calculated by the compress_piece_count_score in the data block header. That is, if the compress_piece_count_score value is an even number, the number of compressed data fragments is:
[0131] N = [compress_piece_count_score / 2], and then according to the number of entries in the compression directory table is 1 more than the number of actual compressed data pieces in the data block, then N+1 is the number of final compression directory entries. However, only each entry corresponding to all compressed data pieces, that is, N entries, is traversed.
[0132] If the compress_piece_count_score value is an odd number, the number of compressed data pieces is
[0133] M = [(compress_piece_count_score-1) / 2], and then according to the number of entries in the compression directory table is 1 more than the number of actual compressed data pieces in the data block, then M+1 is the number of final compression directory entries. However, only each entry corresponding to all compressed data pieces, that is, M entries, is traversed.
[0134] In the above step Step B, the compressed data fragment is decompressed through the external buffer. A data block will only be written by one thread at a time, but there will be multiple threads or processes reading in parallel, so using an external buffer for decompression can ensure parallel reading and is also conducive to caching data.
[0135] Among them, the external buffer is used for decompression during the data reading process. In order to support the maximum concurrency, a buffer of no more than one data block size can be set for each CPU. No matter how much concurrency and user connections the whole system supports, at most only the same number of decompression buffers as the number of CPU cores need to be created for reading. This buffer can be implemented through the thread local storage provided by the operating system.
[0136] In the above step Step B, is there any uncompressed data fragment in the current data block? Which data fragment is the last compressed fragment of the current block? Is the last data fragment stored in the original data area and has not been compressed? Just use a variable in the data block header to find out.
[0137] The value of compress_piece_count_score answers the above three questions:
[0138] Answer to Question 1: The calculation method is based on the parity of the compressed_piece_count_score value; an odd number indicates that there is an uncompressed data fragment; an even number indicates that there is no uncompressed data fragment.
[0139] Answer to Question 2: The calculation method is based on the even value of compress_piece_count_score; compress_piece_count_score / 2 indicates the number and the last compressed piece.
[0140] Answer to Question 3: The calculation method is that the odd value of compress_piece_count_score can be used to determine whether the last data fragment is stored in the original data area and has not been compressed; and (compress_piece_count_score+1) / 2 indicates the number and the last uncompressed data fragment.
[0141] In step B, when reading data from a data block that is being written, if dirty data is read, it is necessary to backtrack and reread. However, dirty data can only be read when reading the original data area. Therefore, before reading data from the original data area, it is necessary to record the status of the currently read data as a transaction point (hereinafter referred to as transaction_begin) so that the original data just read can be deleted for rereading when the reading status is found to be inconsistent.
[0142] In step Step B, before determining that data needs to be read from the original data area, the compress_piece_count_score value in the current block header must be saved. Firstly, after the data reading is completed, the value must be read again and the difference must be compared to determine whether the state of the original data area has changed. Secondly, in case of subsequent data conflicts, the position of the corresponding compressed data area needs to be calculated when the compressed data area is reread.
[0143] Further, after reading from the original data area, you must first
[0144] The parity of the compress_piece_count_score value determines whether there is an uncompressed data segment, and determines the compress_piece_count_score value at this time; then compares it with the compress_piece_count_score value before reading the data from the original data area. If the compress_piece_count_score values in the data block header read twice are different, it means that in the process of just reading the original data, the original data was written into the compression buffer after the data in the compression buffer was compressed. After the compression buffer was allocated to the compression data area, the starting position of the original data area has changed or data belonging to other data segments has been written. The data previously read from the original data area needs to be discarded. Tracing back the previously recorded transaction_begin can speed up the discarding process.
[0145] Among them, "the starting position of the original data area has changed" means that during the data reading process, the position of the original data area is compressed due to writing, and then the original data area is recalculated, so the starting position of the original data area moves to the end of the block and changes. When the data is read, the position of the original data area is closer to the block header than the actual position now, which will definitely cause data errors.
[0146] Furthermore, the subscript calculation method of the original data area to be read in the compressed data area after compression is: (compress_piece_count_score value + 1) / 2. At this time, the recorded compress_piece_count_score value must be an odd number.
[0147] That is, before reading the original data, the compress_piece_count_score value must be an odd number, indicating that there is data in the compressed data area.
[0148] If the compress_piece_count_score value is not equal, it must be re-read from the compressed data area. The compress_piece_count_score value saved before reading the original data + 1 divided by 2 is the table index of the compression directory table. By locating the table entry of the compression directory table, of course, the specific location and size of the compressed data area can be located.
[0149] Therefore, if the verification fails after reading the data from the original data area, the data must be in the compressed data area. After deleting the data that has been read from the original data area, re-read the compressed data from the compressed data area. This process only needs to be repeated once, and there is no need to read it repeatedly. That is: after reading the contents of the original data area, if it is found that a conflict with parallel writes has caused an inconsistent state, it is only necessary to re-read the data in the corresponding compressed area without locking. The backtracking operation only needs to be done once, and there is no need to repeat the detection until the state is consistent. Therefore, it can be concluded that the possibility of backtracking due to a conflict in reading the original data is very small, and because the value of compress_piece_count_score is atomically updated, each backtracking and re-reading action not only does not require locking, but also only needs to be executed once.
[0150] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0151] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data block, characterized in that: include: The data block header, compressed data area, compressed buffer area, original data area and compressed directory table are divided in order from the beginning to the end; The data block header has an atomically modifiable integer variable for storing two states: the number of compressed data fragments and whether there is uncompressed data in the original data area; The compressed data area is used to store compressed data of all data fragments; The compression buffer is configured to: trigger compression when the original data area is full or reaches a preset threshold; the compressed data is stored in the compression buffer; The original data area is configured such that each data value written into the data block by the user is initially concentrated in the area to wait for compression; The compression directory table is used to store the starting position, length and compression type of each compressed data segment in the compressed data area; In a data block, the original data will not be overwritten before the compressed data can be read; and in a data block, all modifications to the written data are in the form of atomic operations.
2. A data block according to claim 1, characterized in that: A data area to be allocated is provided between the data block header and the compressed data area; And / or a data area to be allocated is provided between the compression buffer and the original data area; And / or a data area to be allocated is provided between the original data area and the compressed directory.
3. A data block according to claim 1, characterized in that: The initial value of the integer variable is 0; When the first data in the data block is written into the original data area, the initial value is atomically increased by 1 to become an odd value; When the data in the original data area is compressed and provided for access in the compressed data area, the odd value is atomically increased by 1 to become an even number.
4. A data block according to claim 1, characterized in that: The compression directory table grows from the end of the block to the head of the block.
5. A data block according to claim 1, characterized in that: The number of entries in the compression directory table is one more than the number of actually compressed data segments in the data block; the extra entry is used to reserve space for a newly added compression directory entry.
6. A data block according to claim 1, characterized in that: The storage order of the entries in the compression directory table is in reverse order.
7. A data block according to claim 1, characterized in that: The original data area is further configured to record the number of data items or bytes currently written into the original data area and store them at the end of the original data area.
8. A storage and query method for parallel reading of data blocks, characterized in that: Using the data block according to any one of claims 1 to 7, when writing data, the method comprises: Append the original data written by the user to the original data area of the data block until it is full; Compressing the original data in the original data area and storing the compressed data in a compression buffer in the data block; Allocate the compressed data in the compression buffer to the compressed data area to become a new compressed data fragment; Add a new compressed directory table entry in the compressed directory table to record the information of the new compressed data segment; make the newly generated compressed data area visible to the user; The position and capacity of the original data area and the compression buffer are recalculated in the remaining space of the data block; and the data value to be written is appended to the original data area.
9. A storage and query method for parallel reading of data blocks, characterized in that: Using the data block according to any one of claims 1 to 7, when reading data, the method comprises: Get the number of entries in the compressed directory table in the data block; traverse each entry corresponding to all compressed data fragments; Obtain the position, size and compression type of each compressed data fragment in the compressed data area; Read the compressed data from the compressed data area, and read the original data after decompression in the external buffer; Determine whether the last data fragment of the current data block is in the original data area; If so, record the transaction point where the data has been read, and then read the data from the original data area; After reading the original data, recheck whether the data after the transaction point is valid; If the verification fails, the data after the transaction point needs to be re-read from the compressed data area.
Citation Information
Patent Citations
Method for realizing parallel compression and parallel decompression on FASTQ file containing DNA (deoxyribonucleic acid) sequence read data
CN103559020A
Data storing method and device based on key sorting
CN103870492A
Vehicle sensor data reading device and method
CN110311954A
Data block incremental compression and query method suitable for time sequence database
CN113157680A
Updating method and device of flow information strategy
CN117171460A
