Index structure for persistent storage and data updating and querying method thereof

Abstract

This invention discloses an index structure for persistent storage, belonging to the database field. It includes a deletion flag bit vector and a first and second layer connected by a logical block index. The logical block index is an index constructed from logical blocks, where a logical block is k data blocks obtained by dividing a temporary array containing N data points arranged in a predetermined order. The first layer includes k+1 bit vectors, each containing N bits indicating whether a certain data point in the original array exists in a certain logical block. The second layer includes a sequence of position indicators with N position numbers, representing the position number of the corresponding data in the temporary array within the original array. The deletion flag bit vector includes N bits indicating whether the corresponding data in the original array has been removed. This invention avoids updating the original file's index structure while ensuring the index's validity.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to an index structure for persistent storage and a method for updating and querying data thereon. Background Technology

[0002] In existing disk-based analytical databases, updating the index structure does not delete existing data; instead, it is typically accomplished through restructuring. This renders the index unusable during restructuring, leading to decreased efficiency. Therefore, maintaining the availability of the index structure while simultaneously updating it presents new challenges for the index structure, its data update methods, and query methods. Summary of the Invention

[0003] To address the problem that index functionality cannot be used during index structure reconstruction in existing technologies, the present invention aims to provide an index structure for persistent storage and its data update and query methods, so as to at least partially solve the above-mentioned problems.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] In a first aspect, the present invention provides an index structure for persistent storage, including a deletion marker bit vector and a first layer and a second layer connected by a logical block index;

[0006] The logical block index is an index constructed for the logical block, and the logical block is k data blocks obtained by dividing the temporary array obtained by arranging the original array in a predetermined order.

[0007] The first layer includes k+1 bit vectors, each of which includes N bits, where N is the total number of data in the original array. The bits in the bit vectors are used to indicate whether a certain data in the original array exists in a certain logical block.

[0008] The second layer includes a sequence of position numbers with N position numbers, where each position number represents the position number of the corresponding data in the temporary array within the original array.

[0009] The deletion marker bit vector includes N bits, and the bits in the deletion marker bit vector are used to indicate whether the corresponding data in the original array has been removed, with 1 indicating that it has not been removed and 0 indicating that it has been removed.

[0010] Preferably, the non-empty data in the original array are arranged in ascending order and divided into k equal parts to obtain k logic blocks.

[0011] Preferably, the k+1 bit vectors included in the first layer are denoted as F = {F0, F1, F2, ..., F...} k-1 F k}, where all N bits of bit vector F0 are 0, and bit vector F k All N bits are 1;

[0012] The i-th bit vector F i When the j-th bit in the original array is 1, it indicates that the j-th data in the original array is located in the first i logical blocks; that is, in the i-th bit vector F i In the first i logic blocks, the bits corresponding to the data are all 1, and the remaining bits are all 0, 1≤i≤k, j≥1.

[0013] Preferably, the logical block index is implemented using (value, count) and denoted as S, where S i，value S represents the minimum value in the (i+1)th logic block. i，count This represents the total number of data in the first i logical blocks.

[0014] Secondly, the present invention provides a data update method for an index structure oriented towards persistent storage, the method being applied to the index structure described above, the method comprising the following steps:

[0015] S01. The data to be updated is found in the temporary array using a binary search method;

[0016] S02. Determine the position number of the data to be updated in the original array according to the position number sequence, and set the bit corresponding to the position number in the deletion mark bit vector to 0;

[0017] S03. Append the updated data to the end of the original array, insert the updated data into the temporary array, and update the position label sequence;

[0018] S04. Determine the x-th logical block in which the updated data first appears;

[0019] S05. Append bits 0 to the first x-1 bit vectors in the first layer, and append bits 1 to the xth bit vector and thereafter.

[0020] Thirdly, the present invention provides a data query method for an index structure oriented towards persistent storage, the method being applied to the aforementioned index structure, the method comprising the following steps:

[0021] S1. Determine the query parameters based on the query conditions, and select a bit vector from the first layer as the first layer result based on the query parameters, i.e., bit vector F. tmp ;

[0022] S2. For the bit vector F tmp The correction is performed by inverting the bits that do not meet the query conditions, thereby obtaining the corrected bit vector, which is the query result.

[0023] Preferably, in step S1, a bit vector is selected from the first layer based on the query parameters as the first layer result, i.e., bit vector F. tmp The steps include:

[0024] S11. Determine the logical block where the query parameter is located using the logical block index, denoted as the kth logical block. c A logic block

[0025] S12. Use a binary search method to find the position of the first data that does not meet the query condition in the temporary array, denoted as i. c ;

[0026] S13. The bit vector F tmp Determined by the following formula:

[0027]

[0028] Preferably, in step S2, the bit vector F is... tmp The steps for making corrections include:

[0029] S21. According to k c andi c The data segment in the temporary array that needs to be corrected is identified and denoted as...

[0030] S22. Determine the position of each data point in the original array according to the position label sequence, and then process the bit vector F. tmp The corresponding bits are inverted to obtain the corrected bit vector.

[0031] Furthermore, the method also includes the following steps:

[0032] S3. After obtaining the modified bit vector, merge the modified bit vector with the deletion mark bit vector, and perform a bitwise AND operation on the two to obtain the query result.

[0033] The beneficial effects of the present invention by adopting the above technical solution are as follows: The index structure provided by the present invention, through the deletion mark bit vector and the setting of the first and second layers connected by the logical block index, transforms the data update operation into two steps: deleting the original data and inserting new data. Each data update operation sets the corresponding bit in the deletion mark bit vector of the data to 0, indicating that the data has been deleted, and then inserts the new data. This avoids updating the index structure of the original file and ensures the validity of the index. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the index structure for persistent storage in this invention;

[0035] Figure 2 This is a schematic diagram of the data update process of the index structure in this invention;

[0036] Figure 3 This is a schematic diagram of the data query process of the index structure in this invention. Detailed Implementation

[0037] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "front", "rear", etc., indicate the orientation or positional relationship based on the description of the structure of this invention shown in the accompanying drawings. They are only for the convenience of describing this invention and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0039] The terms "first" and "second" in this technical solution are merely designations for corresponding structures that are identical or similar, or that perform similar functions. They do not represent an arrangement of the importance of these structures, nor do they imply any ranking, comparison of size, or other meaning.

[0040] Furthermore, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two structures. Those skilled in the art can understand the specific meaning of the above terms in this invention by considering the overall concept of the invention and the specific context of the solution.

[0041] Example 1

[0042] An index structure for persistent storage includes a deletion marker bit vector and a first and second layer connected by logical block indexes.

[0043] The logical block index refers to the index built on the logical block. The logical block is the k data blocks obtained by dividing the temporary array obtained by arranging the original array in a predetermined order (e.g., ascending order). The logical block only represents the logical division of the data and does not need to be stored separately. Therefore, the data blocks obtained by the division are called logical blocks.

[0044] In this embodiment, the non-empty data (e.g., E empty data) in the original array (e.g., N arrays) are arranged in ascending order and divided into k equal parts to obtain k logic blocks. Each logic block contains (NE) / K data, denoted as A = {A1, A2, ..., A...}. K}, where N is the total number of original data, and the k logical blocks are divided to satisfy the following conditions:

[0045] A imin ≤A imax ≤A (i+1)min ≤A (i+1)max

[0046] A imin and A imax Representing logic block A respectively i The minimum and maximum values ​​in A (i+1)min and A (i+1)max Representing logic block A respectively i+1 The minimum and maximum values ​​in.

[0047] The first layer includes k+1 bit vectors, each containing N bits. That is, the number of bits in the bit vector is the same as the total number of data in the original array. The bits in the bit vector are used to indicate whether a certain data in the original array exists in a certain logical block.

[0048] In this embodiment, the k+1 bit vectors included in the first layer are denoted as F = {F0, F1, F2, ..., F...} k-1 F k}, where all N bits of bit vector F0 are 0, and bit vector F k All N bits are 1. The i-th bit vector F i When the j-th bit in the array is 1, it means that the j-th data in the original array is located in the first i logical blocks, i.e. In the middle. Conversely, in the i-th bit vector F iIn the first i logic blocks, the bits corresponding to the data are all 1, and the remaining bits are all 0, 1≤i≤k, j≥1.

[0049] The second layer includes a sequence of position labels with N position numbers, where each position number represents the position number of the corresponding data in the temporary array within the original array.

[0050] The deletion flag vector also includes N (i.e. 16) bits. The bits in the deletion flag vector are used to indicate whether the corresponding data in the original array has been removed. When a data is deleted, the corresponding bit in the deletion flag vector is set to 0, otherwise it is set to 1.

[0051] like Figure 1 As shown, for example, the original array is denoted as data B, which contains 16 data. After sorting data B in ascending order, a temporary array is obtained. After dividing the temporary array into 4 equal parts (i.e., k=4), four logic blocks A1-A4 are obtained, denoted as A={A1, A2, A3, A4}, and each logic block contains 4 data.

[0052] Logical block indexes are implemented using (value, count) and denoted as S. Typically, indexes are built for each logical block starting from the second logical block. i，value S represents the minimum value in the (i+1)th logic block. i，count This represents the total number of data in the first i logical blocks. For example, (114, 8) represents S. 2，value =114, S 2，count =8, meaning the minimum value in the third logic block is 114, and the sum of the data in the first two logic blocks is 8.

[0053] The first layer contains 5 bit vectors, where bit vector F0 has all 16 bits set to 0, and bit vector F5 has all 16 bits set to 1. Therefore, in this embodiment, bit vectors F0 and F5 are not stored. The three bit vectors stored in the first layer are denoted as F = {F1, F2, F3}. For details on the three bit vectors stored in the first layer, please refer to [link / reference]. Figure 1 .

[0054] Second-level storage, such as Figure 1 The set of 16 data points shown represents a position number, which is the position number of the corresponding data in the original data in the temporary array. For example, the first data point "8" in the temporary array is the third data point in the original data, so it corresponds to the data "3" in the position number sequence.

[0055] For null values ​​in the original array, the corresponding bits in all bit vectors of the first level are set to 0. The logical block index can be constructed based on any data type, meaning it is independent of the specific data type but related to the relative size of the data. Logical block indices can be obtained from comparable data, thus constructing a two-level index structure.

[0056] Example 2

[0057] A data update method for an index structure oriented towards persistent storage, applied to the index structure disclosed in Embodiment 1, includes the following steps:

[0058] S01. Find the data to be updated in the temporary array using a binary search method;

[0059] For example, a data update instance of "20 => 91" means deleting the data "20" from the original array and then adding the data "91" to the original array. Then, as follows... Figure 2 As shown, the data "20" is located in the second position of the temporary array, and its corresponding position number in the position label sequence is "10", as follows. Figure 2 As shown in step "①".

[0060] S02. Determine the position number of the data to be updated in the original array according to the position number sequence, and set the bit corresponding to the position number in the deletion mark bit vector to 0;

[0061] The position index sequence determines that the data "20" is at position "10" in the original array. Therefore, the 10th bit in the deletion marker bit vector is set to 0. Figure 2 As shown in step "②".

[0062] S03. Append the updated data to the end of the original array, insert the updated data into the temporary array, and update the position label sequence;

[0063] Append the data "91" to the end of the original data, such as... Figure 2 As shown in step "③". By comparing the sizes, it can be determined that the data "91" is between the data "74" and the data "114". Therefore, the data "91" belongs to logic block A2 and is the maximum value of logic block A2. It corresponds to the eighth position in the position number sequence, and its position number is updated to "16", as shown. Figure 2 As shown in step “④”.

[0064] S04. Determine the x-th logical block where the update data first appears;

[0065] S05. Append bits 0 to the first x-1 bit vectors in the first layer, and append bits 1 to the xth bit vector and thereafter.

[0066] As shown in step S03, the data "91" first appears in logic block A2. Therefore, it is necessary to append 1 to the 16th bit of the bit vector from the 2nd bit onwards in the second layer, as follows: Figure 2 As shown in step "⑤".

[0067] At this point, the index structure has completed the data update operation of "20 => 91". Furthermore, in this embodiment of the invention, the data update operation is converted into two steps: deleting the original data and inserting new data. Each data update operation sets the corresponding bit in the deletion flag bit vector of the data to 0, indicating that the data has been deleted, and then inserts the new data. This avoids updating the index structure of the original file and also ensures the validity of the index.

[0068] Example 3

[0069] A data query method for an index structure oriented towards persistent storage, applied to the index structure disclosed in Embodiment 1, includes the following steps:

[0070] S1. Determine the query parameters based on the query conditions, and select a bit vector from the first layer as the first layer result based on the query parameters, i.e., bit vector F. tmp ;

[0071] S2. For positional vector F tmp The correction is performed by inverting the bits that do not meet the query conditions, thus obtaining the corrected bit vector, which is the query result.

[0072] In step S1, a bit vector is selected from the first layer based on the query parameters as the first layer result, i.e., bit vector F. tmp The steps include:

[0073] S11. Determine the logical block containing the query parameters using the logical block index, denoted as the k-th logical block. c A logic block

[0074] For example, such as Figure 3 As shown, when the query condition is set to "x < 69", the corresponding query parameter is "69". The logical block index can be used to locate that "69" belongs to the interval [S]. 1，value S 2，value The interval is [52, 114), therefore "69" belongs to the second logical block A2, i.e., k. c =2, such as Figure 3 As shown in step "①".

[0075] S12. Use binary search to find the position of the first data that does not meet the query condition in the temporary array, denoted as i. c ;

[0076] As can be seen, the first data that does not meet the query criteria is located at position "7", i. c =7.

[0077] S13. Position vector F tmp Determined by the following formula:

[0078]

[0079] will i c Substituting 7 into the above formula, we can obtain F. tmp For F2, such as Figure 3 As shown in step "②". However, in bit vector F2, the bits corresponding to the data in the interval [8, 114) are all 1, while this embodiment requires querying "x < 69", so the bits corresponding to the data in the interval [69, 114) should be 0. Therefore, bit vector F2 needs to be corrected.

[0080] In step S2, the bit vector F tmp The steps for making corrections include:

[0081] S21. According to k c andi c Identify the data segment in the temporary array that needs correction, denoted as .

[0082] The sum of data S of the first two logic blocks 2，count =8, the first data that does not meet the query condition is located at position "7", that is, the bits corresponding to the data with the rank number in the interval [7, 8] in the temporary array need to be corrected, that is, the bits corresponding to the data "74" and data "91" in the bit vector F2 need to be corrected.

[0083] S22. Determine the position of each data point in the original array based on the position label sequence, and then adjust the bit vector F. tmp The corresponding bits are inverted to obtain the corrected bit vector.

[0084] Based on the position number sequence, it can be seen that the position number corresponding to the data "74" in the temporary array is "5", and the position number corresponding to the data "91" in the temporary array is "16". Figure 3 As shown in step “③”.

[0085] That is, the 5th and 16th bits in bit vector F2 need to be corrected. This can be done by inverting the 5th and 16th bits in bit vector F2. The bit vector F2 after inversion and correction is the query result.

[0086] In this embodiment, for bit vector Ftmp When making corrections, memory prefetching is used to load the bits that need to be corrected into the cache, thereby minimizing the overhead of random memory access.

[0087] Example 4

[0088] Based on the method disclosed in Embodiment 3, this method further includes the following steps:

[0089] S3. After obtaining the corrected bit vector, merge the corrected bit vector with the deletion mark bit vector, and perform a bitwise AND operation on the two to obtain the query result.

[0090] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A data update method for an index structure oriented towards persistent storage, characterized in that: The index structure for persistent storage includes a deletion marker bit vector and a first and second layer connected by logical block indexes; The logical block index is an index constructed for the logical block, and the logical block is k data blocks obtained by dividing the temporary array obtained by arranging the original array in a predetermined order. The first layer includes k+1 bit vectors, each of which includes N bits, where N is the total number of data in the original array. The bits in the bit vectors are used to indicate whether a certain data in the original array exists in a certain logical block. The second layer includes a sequence of position numbers with N position numbers, where each position number represents the position number of the corresponding data in the temporary array within the original array. The deletion marker bit vector includes N bits. The bits in the deletion marker bit vector are used to indicate whether the corresponding data in the original array has been removed, with 1 indicating that it has not been removed and 0 indicating that it has been removed. The method includes the following steps: S01. The data to be updated is found in the temporary array using a binary search method; S02. Determine the position number of the data to be updated in the original array according to the position number sequence, and set the bit corresponding to the position number in the deletion mark bit vector to 0; S03. Append the updated data to the end of the original array, insert the updated data into the temporary array, and update the position label sequence; S04. Determine the x-th logical block in which the updated data first appears; S05. Append bits 0 to the first x-1 bit vectors in the first layer, and append bits 1 to the xth bit vector and thereafter.

2. A data query method for an index structure oriented towards persistent storage, characterized in that: The index structure for persistent storage includes a deletion marker bit vector and a first and second layer connected by logical block indexes; The logical block index is an index constructed for the logical block, and the logical block is k data blocks obtained by dividing the temporary array obtained by arranging the original array in a predetermined order. The first layer includes k+1 bit vectors, each of which includes N bits, where N is the total number of data in the original array. The bits in the bit vectors are used to indicate whether a certain data in the original array exists in a certain logical block. The second layer includes a sequence of position numbers with N position numbers, where each position number represents the position number of the corresponding data in the temporary array within the original array. The deletion marker bit vector includes N bits. The bits in the deletion marker bit vector are used to indicate whether the corresponding data in the original array has been removed, with 1 indicating that it has not been removed and 0 indicating that it has been removed. The method includes the following steps: S1. Determine the query parameters based on the query conditions, and select a bit vector from the first layer as the first layer result based on the query parameters, i.e., a bit vector. ; S2. For the bit vector The correction is performed by inverting the bits that do not meet the query conditions, thereby obtaining the corrected bit vector, which is the query result.

3. The method according to claim 2, characterized in that: In step S1, a bit vector is selected from the first layer based on the query parameters as the first layer result, i.e., a bit vector. The steps include: S11. Determine the logical block where the query parameter is located using the logical block index, denoted as the first... A logic block ; S12. Use a binary search method to find the position of the first data that does not meet the query condition in the temporary array, denoted as . ; S13. The bit vector Determined by the following formula: 。 4. The method according to claim 3, characterized in that: In step S2, the bit vector The steps for making corrections include: S21. According to and The data segment in the temporary array that needs to be corrected is identified and denoted as... ; S22. Determine the position of each data point in the original array according to the position label sequence, and then process the bit vector. The corresponding bits are inverted to obtain the corrected bit vector.

5. The method according to claim 2, characterized in that: The method further includes the following steps: S3. After obtaining the modified bit vector, perform an AND operation on the modified bit vector and the deletion marker bit vector to merge them and obtain the query result.

6. The method according to claim 1 or 2, characterized in that: Sort the non-empty values ​​in the original array in ascending order and then... After dividing into equal parts, we get The aforementioned logical blocks.

7. The method according to claim 1 or 2, characterized in that: The first layer includes The bit vectors are denoted as , where bit vector All N bits are 0, bit vector All N bits are 1; No. Units vector The first in When the nth bit is 1, it indicates that the nth bit in the original array is 1. The data is located in the first i logical blocks; that is, in the i-th logical block... Units vector In the first i logic blocks, all bits corresponding to the data are 1, and the remaining bits are 0. , .

8. The method according to claim 1 or 2, characterized in that: The logical block index is implemented using (value, count) and denoted as S, where, Indicates the first The minimum value in each logical block This represents the total number of data in the first i logical blocks.

Images