Index compression method, device, computing device cluster and storage medium

Abstract

The present application discloses an index compression method, device, computing device cluster and storage medium, which belong to the field of database. The method includes: obtaining a tree index structure in a database, the tree index structure includes at least two layers of index nodes; scanning index nodes in at least two layers of index nodes; when the scanned first index node meets the selection condition, compressing the first index node to obtain a compressed package. The present application selects the first index node in the tree index structure according to the selection condition, and compressing the first index node can improve the overall compression rate of the database. Moreover, by selective compression, rather than unselectively compressing multiple index nodes in full, the compression effect is guaranteed, and compression of meaningless index nodes is avoided.

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular to an index compression method, device, computing device cluster and storage medium. Background Art

[0002] In a database, indexes are usually used to speed up data queries.

[0003] The related art provides a B-Tree index, which is a multi-branch balanced tree. The B-Tree index consists of at least two layers of index nodes, each of which stores multiple index keywords. When querying, based on a given index keyword, a binary search is performed layer by layer starting from the top layer of at least two layers of index nodes in the B-Tree index until the index node corresponding to the given index keyword is found.

[0004] In actual application scenarios, index pages (index nodes) also occupy a certain amount of storage space. Summary of the invention

[0005] The present application provides an index compression method, device, computing device cluster and storage medium, which can improve the overall compression rate of the database, reduce the size of the database, and ensure the compression effect through selective compression. The technical solution is as follows.

[0006] In a first aspect, an index compression method is provided, the method comprising: obtaining a tree-like index structure in a database, the tree-like index structure comprising at least two layers of index nodes; scanning index nodes in at least two layers of index nodes; and compressing the first index node to obtain a compressed package when the scanned first index node meets a selection condition.

[0007] The present application will select the first index node in the tree index structure according to the selection conditions. Compressing the first index node can improve the overall compression rate of the database. Moreover, by selectively compressing, rather than compressing multiple index nodes in full without selection, the compression effect is guaranteed and meaningless index nodes are avoided from being compressed. Moreover, the index compression method provided by the present application can be applied to any tree index structure, for example, it can be applied to both B-tree index and B+ tree index.

[0008] Optionally, the first index node satisfies a selection condition, including at least one of the following:

[0009] The first index node is an index node for indexing cold data in the database, where cold data refers to data whose access heat is less than a heat threshold; optionally, the cold data includes at least one of the following: data in the database that has not been accessed within a first time period; data in the database whose access frequency is less than a frequency threshold.

[0010] This application chooses to compress the index nodes corresponding to the cold data to avoid frequent decompression operations during subsequent business queries, because the cold data will not be frequently accessed. After the compression is completed, the index nodes corresponding to the cold data only need to be decompressed in a few cases where the cold data is accessed.

[0011] The first index node is an index node used to index low-reuse data in the database, where the low-reuse data refers to data shared by services that does not exceed a quantity threshold.

[0012] If there is some data in the database that is shared by more businesses, it means that the data is more likely to be accessed again, and if there is some data in the database that is shared by fewer businesses, it means that the data is more likely to be cold data. In this application, compressing the index to the index node of low-reuse data in the database can effectively avoid frequent decompression operations later.

[0013] The first index node is an index node whose expected compression rate exceeds the compression rate threshold.

[0014] This application will compress index nodes whose expected compression rate exceeds the compression rate threshold, which can improve the use effect of compression resources, including computing resources for performing compression operations and storage resources for storing compressed packages. In the case of limited compression resources, only compression resources are consumed on index nodes with good expected compression effects, and compression resources are not wasted on index nodes with poor expected compression effects, which can make full use of limited compression resources and improve the use effect of compression resources.

[0015] Optionally, the method further includes: storing the compressed package in a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure.

[0016] The present application stores the compressed package in the sibling index node of the first index node, and there is no need to set up an additional storage location to store the compressed package. The method of storing the compressed package provided by the present application can avoid the overhead of additional storage space.

[0017] Optionally, the method further includes: storing the meta information of the compressed package in the storage space of the second index node.

[0018] For the compressed package, the present application stores the metadata of the compressed package in the storage space of the second index node. The data volume of the metadata is smaller than the data volume of the entire compressed package. Compared with storing the entire compressed package, the present application stores the metadata of the compressed package in the storage space of the second index node to avoid excessive occupation of the storage space of the second index node. Moreover, if the compressed package is large in size, the advantage of storing only the metadata of the compressed package in the storage space of the second index node is more obvious.

[0019] Optionally, the method further includes: storing the compressed package in a newly created independent file.

[0020] This application stores the compressed package in a newly created independent file to avoid occupying the storage space of other index nodes in the tree index structure. By storing the compressed package in a newly created independent file, the compressed package can be placed independently, which is convenient for the management of the compressed package.

[0021] Optionally, the method further includes: storing the metadata of the compressed package in a newly created independent file.

[0022] For the compressed package, this application stores the metadata of the compressed package in a newly created independent file, which is convenient for unified management of the compressed package. For example, when scanning and selecting a large number of index nodes, the metadata of multiple compressed packages corresponding to multiple index nodes are uniformly stored in a newly created independent file, which can realize unified management of multiple compressed packages.

[0023] Optionally, the method further includes: reclaiming storage space corresponding to the first index node.

[0024] In addition to compressing the selected first index node, the present application also actively reclaims the storage space of the first index node, thereby improving the utilization rate of the index node, and the reclaimed storage space can be used for other transactions.

[0025] Optionally, after compressing the first index node to obtain a compressed package, it also includes: using the next number of the assigned transaction number as the recycle number of the first index node; reclaiming the storage space corresponding to the first index node includes: scanning the index nodes in at least two layers of index nodes again; when the first index node is scanned, if the recycle number of the first index node is less than the minimum number of the currently active transaction, reclaiming the storage space corresponding to the first index node.

[0026] This application takes into account the situation that the storage space of the first index node cannot be reclaimed immediately after compression. By assigning a latest transaction number as the recycle number of the first index node after compression, the relationship between the recycle number and the current smallest active transaction number will be judged during recycle. This can ensure that no other transactions access the first index node after recycle, thereby realizing active recycle of the storage space of the index node and ensuring the normal execution of other transactions.

[0027] Optionally, the tree index structure is a B+ tree index; scanning the index nodes in at least two layers of index nodes includes: in the last layer of index nodes of at least two layers of index nodes, scanning the index nodes one by one according to a reference order, and the reference order includes the order of index keywords existing in the index nodes from large to small, or from small to large.

[0028] The present application will scan the last layer of index nodes to select the index nodes to be compressed. For the B+ tree index, only the last layer of index nodes stores the index keywords, and only the last layer of index nodes in the B+ tree index stores the target access data or the access address of the target access data, that is, when querying through the B+ tree index, the index nodes of the last layer will inevitably be queried. At this time, only scanning and compressing the index nodes of the last layer will ensure the correctness of the compression, and will not compress the index nodes that do not carry valid data, thereby avoiding meaningless compression operations and wasting computing resources and storage resources.

[0029] Optionally, the first index node stores any one of the following data: an index keyword and target access data, where the target access data refers to data accessed when querying the tree index structure; and an access address of the index keyword and the target access data.

[0030] For a tree-like index structure, all or part of the index nodes may carry index keywords and target access data, or may carry index keywords and access addresses of target access data. The index compression method provided in this application is applicable to both cases.

[0031] Optionally, the method further includes: in the process of querying the tree index structure, if a first index node is hit, obtaining a compressed package of the first index node; decompressing the compressed package to restore internal data of the first index node.

[0032] In the process of querying the tree index structure, if the first index node is hit, the present application only needs to decompress the first index node, and does not need to decompress other irrelevant index nodes. The performance overhead required by the present application is relatively small.

[0033] Optionally, obtaining the compressed package of the first index node includes: obtaining the compressed package of the first index node from a storage space of a second index node, the second index node being a sibling index node of the first index node in the tree index structure.

[0034] Based on the characteristics of the tree index structure, when querying the tree index structure, the query will be performed layer by layer according to the index keywords. If the current query needs to obtain the first index node corresponding to the first keyword interval, and the first index node has been compressed, based on the agreed storage location rules, it will be known to obtain the compressed package of the first index node from the second index node corresponding to the keyword interval adjacent to the first keyword interval (the second keyword interval). At this time, utilizing the characteristics of the tree index structure and the agreed storage location rules, the present application can obtain the compressed package of the first index node faster.

[0035] Optionally, obtaining the compressed package of the first index node includes: obtaining meta information of the compressed package of the first index node from the storage space of the second index node; and obtaining the compressed package of the first index node according to an acquisition position indicated by the meta information.

[0036] The present application stores the metadata of the compressed package in the sibling index nodes of the first index node, and utilizes the characteristics of the tree index structure and the agreed storage location rules to obtain the metadata of the compressed package of the first index node more quickly, thereby obtaining the compressed package more quickly.

[0037] Optionally, obtaining the compressed package of the first index node includes: obtaining the compressed package of the first index node from a newly created independent file.

[0038] When querying the tree index structure, if the first index node is hit, the present application will obtain the compressed package of the first index node from the newly created independent file. When obtaining, there is no need to distinguish the compressed package in the storage space shared with other data. For example, the storage location of the compressed package is determined based on the offset address in the storage space shared with other data.

[0039] Optionally, obtaining the compressed package of the first index node includes: obtaining meta information of the compressed package of the first index node from a newly created independent file; and obtaining the compressed package of the first index node according to an acquisition position indicated by the meta information.

[0040] The present application implements hierarchical query when querying compressed packages, which is not prone to errors. For example, in the process of querying the tree index structure, each time a compressed index node is queried, the metadata of the compressed package will be uniformly obtained from the newly created independent file, and then the compressed package will be further obtained based on the acquisition position indicated by the metadata, thus realizing hierarchical query of the compressed package.

[0041] In a second aspect, an index compression device is provided, wherein the index compression device has the function of implementing the method described in the first aspect. The index compression device includes at least one module, and the at least one module is used to implement the method described in the first aspect. The at least one module includes:

[0042] An acquisition module is used to acquire a tree index structure in a database, where the tree index structure includes at least two layers of index nodes;

[0043] A scanning module, used for scanning index nodes in at least two layers of index nodes;

[0044] The compression module is used to compress the first index node to obtain a compressed package when the scanned first index node meets the selection condition.

[0045] Optionally, the first index node satisfies a selection condition, including at least one of the following:

[0046] The first index node is an index node for indexing cold data in the database, where cold data refers to data whose access heat is less than a heat threshold; optionally, the cold data includes at least one of the following data: data in the database that has not been accessed within a first time period; data in the database whose access frequency is less than a frequency threshold.

[0047] The first index node is an index node for indexing low-reuse data in the database, where the low-reuse data refers to data shared by services that does not exceed a quantity threshold;

[0048] The first index node is an index node whose expected compression rate exceeds the compression rate threshold.

[0049] Optionally, the device further comprises a storage module, the storage module being used to store the compressed package in a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure.

[0050] Optionally, the device further includes a storage module, and the storage module is used to store the meta information of the compressed package in the storage space of the second index node.

[0051] Optionally, the device further comprises a storage module, and the storage module is used to store the compressed package in a newly created independent file.

[0052] Optionally, the device further comprises a storage module, and the storage module is used to store the metadata of the compressed package in a newly created independent file.

[0053] Optionally, the device further includes a recycling module, which is used to recycle the storage space corresponding to the first index node.

[0054] Optionally, the device also includes a recycling module, which is used to use the next number of the assigned transaction number as the recycling number of the first index node; scan the index nodes in at least two layers of index nodes again; when the first index node is scanned, if the recycling number of the first index node is less than the minimum number of the currently active transaction, recycle the storage space corresponding to the first index node.

[0055] Optionally, the tree index structure is a B+ tree index; the scanning module is used to scan the index nodes in the last layer of index nodes of at least two layers of index nodes according to a reference order, and the reference order includes the order of index keywords existing in the index nodes from large to small, or from small to large.

[0056] Optionally, the first index node stores any one of the following data: an index keyword and target access data, where the target access data refers to data accessed when querying the tree index structure; and an access address of the index keyword and the target access data.

[0057] Optionally, the device further comprises a query module, and the query module is used to obtain a compressed package of the first index node if a first index node is hit during a query of the tree index structure; decompress the compressed package to restore internal data of the first index node.

[0058] Optionally, the query module is further used to obtain a compressed package of the first index node from a storage space of a second index node, and the second index node is a sibling index node of the first index node in the tree index structure.

[0059] Optionally, the query module is further used to obtain meta information of the compressed package of the first index node from the storage space of the second index node; and obtain the compressed package of the first index node according to the acquisition position indicated by the meta information.

[0060] Optionally, the query module is further used to obtain a compressed package of the first index node from the newly created independent file.

[0061] Optionally, the query module is further used to obtain meta information of the compressed package of the first index node from the newly created independent file; and obtain the compressed package of the first index node according to the acquisition position indicated by the meta information.

[0062] In a third aspect, a computing device is provided, the computing device comprising a processor, the processor being coupled to a memory, the memory being used to store a computer program for executing the method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the method provided in the first aspect.

[0063] Optionally, the computing device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0064] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor, the processor being coupled to a memory;

[0065] The processor of the at least one computing device is used to execute instructions stored in the memory, so that the computing device cluster executes the method described in the first aspect above.

[0066] In a fifth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0067] In a sixth aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0068] The technical effects obtained in the above-mentioned second to sixth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0070] Figure 2 is a flowchart of an index compression method provided by an embodiment of the present application;

[0071] Figure 3 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0072] Figure 4 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0073] Figure 5 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0074] Figure 6 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0075] Figure 7 is a schematic diagram of an index compression method provided in an embodiment of the present application;

[0076] Figure 8 is a flowchart of an index compression method provided by an embodiment of the present application;

[0077] Fig. 9 is a structural schematic diagram of an index compression device provided in an embodiment of the present application;

[0078] Fig.10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0079] Fig.11 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0080] Fig.12 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0082] First, some nouns involved in the embodiments of this application are briefly introduced:

[0083] A tree index structure refers to a distributed storage data structure created to speed up the retrieval of data. The tree index structure can speed up the speed at which users query data in a database. There are at least two layers of index nodes in the tree index structure, and the at least two layers of index nodes include a root index node layer and a leaf index node layer. Optionally, the tree index structure also includes an intermediate index node layer located between the root index node layer and the leaf index node layer. There is at least one index node in each index node layer. The index node is the physical storage unit of the index. The index node is often called an index block or index page. Normally, the size of the index node is 8KB or 16KB. In the tree index structure, index nodes in the same layer have a brother relationship, and index nodes in the same branch in the upper and lower layers have a parent-child relationship.

[0084] Common tree-like index structures include B-tree index and B+-tree index. B-tree index includes at least two layers of index nodes, each of which stores at least one index keyword. When querying, binary search will be performed layer by layer based on the given index keyword, starting from the top layer. When the given index keyword is the same as the index keyword stored in the current index node, the query ends, otherwise the query will continue to query the next layer of index nodes. Each index node of the B-tree index also stores the specific data corresponding to its own index keyword or the access address of the data.

[0085] The B+ tree index also includes at least two layers of index nodes. The difference from the B-tree index is that in the B+ tree index, all index keywords are stored in the last layer of index nodes, and the middle layer of index nodes only stores the index of the index keyword, for example, the maximum value of all index keywords in the corresponding child index node. When querying, based on the given index keyword, binary search will be performed layer by layer starting from the top layer and ending at the bottom layer. In the B+ tree index, only the bottom layer of index nodes stores specific data or the access address of the data.

[0086] The index compression method provided in this application is applicable to the tree index structure in the database.

[0087] Figure 1 A schematic diagram of an index compression method provided by an exemplary embodiment of the present application is shown.

[0088] Figure 1 Part (A) shows a two-layer tree index structure, the first index node layer includes index node 1, and the second index node layer includes index nodes 2, 3, and 4. Assuming that index node 3 is selected during the process of scanning index nodes one by one, the present application will compress the internal data of index node 3. Figure 1As shown in part (B), a compressed package is obtained by compressing the internal data of index node 3, and the compressed package is stored in the storage space of index node 4. Index node 4 is an adjacent sibling index node of index node 3. At the same time, optionally, index nodes 2 and 4 are also allowed to be adjacent sibling index nodes to each other. For example, the page number of index node 4 is added as the page number of the sibling index node in the storage space of index node 2, and the page number of index node 2 is added as the page number of the sibling index node in the storage space of index node 4. Optionally, the page number of index node 3 as a sibling index node is also deleted from the storage space of index nodes 2 and 4.

[0089] Due to the existence of concurrent transactions, after index node 3 is compressed, other transactions may access index node 3. Therefore, the storage space of index node 3 will not be reclaimed immediately, but will be reclaimed at a later appropriate time. Figure 1 As shown in part (C), when no other transaction will access index node 3, the storage space of index node 3 is reclaimed. Reclaiming the storage space of index node 3 means clearing all data in the storage space of index node 3, so that the storage space becomes a new available storage space.

[0090] Figure 2 is a flowchart of an index compression method provided in an embodiment of the present application. The method can be applied to any computing device or a computing device cluster including multiple computing devices. Figure 2 The method shown includes the following steps.

[0091] Step 220: Obtain the tree index structure in the database.

[0092] There is a tree index structure in the database. The tree index structure is a decentralized storage data structure. Users can quickly query the target access data based on the tree index structure. There are at least two layers of index nodes in the tree index structure, and each layer includes at least one index node. The index node is often called an index block or index page. The index node is the physical storage unit of the index. When querying, one index node will be loaded at a time. The size of each index node is fixed, usually 8KB or 16KB. In one embodiment, the tree index structure in the database is loaded into the memory of the computing device to facilitate the computing device to scan the index nodes in the tree index structure.

[0093] Optionally, the tree index structure includes any one of a B-tree index and a B+ tree index.

[0094] Optionally, the database can be any one of a MySQL database, an Oracle database, a SQL Server database, a Sybase database, a DB2 database, a MongoDB database, a Redis database, a PostgreSQL database, and a SQLite database. The index compression method provided in the present application is applicable to any database, so the method provided in the present application is universal.

[0095] Step 240: Scan the index nodes in at least two levels of index nodes.

[0096] In one embodiment, in the last layer of index nodes of at least two layers of index nodes, each index node is scanned one by one. If the tree index structure is a B+ tree index, only the last layer of index nodes in the B+ tree index stores index keywords, and only the last layer of index nodes in the B+ tree index stores target access data or the access address of target access data, that is, when querying through the B+ tree index, the last layer of index nodes will inevitably be queried. At this time, only the last layer of index nodes is scanned and compressed, which will ensure the correctness of compression, and will not compress index nodes that do not carry valid data, avoiding the execution of meaningless compression operations and wasting computing resources and storage resources.

[0097] Optionally, each index node in the last layer of index nodes is scanned one by one according to the reference order. The reference order includes the order of index keywords existing in the index nodes from large to small, or from small to large. In the tree index structure, the index nodes of each layer are arranged in order based on the size of the index keywords. The present application also performs scanning based on the size of the index keywords, which is consistent with the distribution of the tree index structure itself. It is not easy to make mistakes during scanning, and it is not easy to miss or repeat the scan.

[0098] In one embodiment, each index node in the tree index structure is scanned, and each index node in the tree index structure is scanned one by one. If the tree index structure is a B-tree index, the index nodes at each level in the B-tree index store index keywords, as well as target access data or access addresses of target access data corresponding to the index keywords. That is, when querying through the B-tree index, the query may be stopped at any index node at any level, and the index nodes at the last level may not be queried. At this time, each index node in the tree index structure is scanned, which can ensure that the target index node is selected during the scan, and the target index node stores the target access data to be compressed or the access address of the target access data.

[0099] Optionally, the index nodes are scanned one by one in a reference order in the tree index structure. The reference order includes the following two.

[0100] The first possible scanning order: scan layer by layer from the top to the bottom, and scan the index nodes one by one in each layer according to the order of index keywords from large to small, or from small to large. The second possible scanning order: scan layer by layer from the bottom to the top, and scan the index nodes one by one in each layer according to the order of index keywords from large to small, or from small to large. The scanning order adopted by the present application conforms to the distribution of the tree index structure itself, and is not prone to errors during scanning, and is not prone to missed scanning or repeated scanning.

[0101] Step 260: When the scanned first index node meets the selection condition, compress the first index node to obtain a compressed package.

[0102] In one embodiment, the tree index structure is a B+ tree index. In the index node of the last layer of the B+ tree index, the internal data of the index node includes the index keyword and the target access data, or the internal data of the index node includes the index keyword and the access address of the target access data. The target access data refers to the data accessed when querying the tree index structure. When compressing, all the internal data of the index node are compressed to obtain a compressed package.

[0103] In one embodiment, the tree index structure is a B-tree index, and the internal data of each index node in the B-tree index includes an index keyword, and target access data corresponding to the index keyword or an access address of the target access data. The target access data refers to the data accessed when querying the tree index structure. In addition, except for the index nodes of the last layer, the internal data of the index nodes of other layers in the B-tree index also include the page numbers of the corresponding child index nodes, so as to facilitate loading the lower-layer child index nodes when querying. When compressing, all the internal data of an index node will be compressed to obtain a compressed package.

[0104] In the present application, the internal data of the index nodes that meet the selection conditions will be compressed, and the selection conditions will be described in detail below. The present application will select the first index node in the tree index structure according to the selection conditions. Compressing the first index node can improve the overall compression rate of the database. Moreover, by selective compression rather than unselectively compressing multiple index nodes in full, the compression effect is guaranteed and the compression of meaningless index nodes is avoided. Moreover, the index compression method provided by the present application can be applied to any tree index structure, for example, it can be applied to both B-tree index and B+ tree index.

[0105] In the present application, the compression algorithm may adopt a general compression algorithm, such as RLE algorithm, Huffman algorithm, Rice algorithm, etc., which are all applicable, and the present application does not impose any limitation on this.

[0106] In one embodiment, when querying the tree index structure later, if the compressed package of the first index node is frequently decompressed, the internal data of the first index node is restored so that the first index node can be directly queried when querying the tree index structure. Optionally, if the number of times the compressed package of the first index node is decompressed exceeds a decompression number threshold within a preset time length, it is determined that the compressed package of the first index node is frequently decompressed.

[0107] In one embodiment, the preset duration is a time length set by a technician, or the preset duration is determined based on a computing device that performs the decompression operation. For example, if a computing device with poor performance is used to perform the decompression operation, the preset duration is set to a first time length, and if a computing device with better performance is used to perform the decompression operation, the preset duration is set to a second time length, and the first time length is less than the second time length. If a computing device with better performance is used to perform the decompression operation, a longer time can be used to determine whether the compression of the first index node is frequently decompressed, and the judgment result is more accurate at this time. If a computing device with poor performance is used to perform the decompression operation, a shorter time can be used to determine whether the compression of the first index node is frequently decompressed, which can reduce the performance overhead of the computing device.

[0108] In one embodiment, when querying the tree index structure, if the first index node is hit, the compressed package of the first index node is obtained; the compressed package is decompressed to restore the internal data of the first index node. When querying the tree index structure, the query is performed based on the index keyword. If the first index node is hit by the index keyword, but the first index node has been compressed, the compressed package of the first index node is obtained, the compressed package is decompressed, and the internal data of the first index node is restored.

[0109] In the present application, in the process of querying the tree index structure, the present application only needs to decompress the selected first index node, and does not need to decompress other irrelevant index nodes. The performance overhead required by the present application is relatively small.

[0110] In the present application, the index nodes are scanned, and the internal data of the first index node that meets the selection condition is compressed. In one embodiment, the first index node meets the selection condition, including at least one of the following:

[0111] 1. The first index node is an index node used to index cold data in the database;

[0112] Cold data refers to data whose access heat is less than the heat threshold. Cold data is data whose access heat is less than the heat threshold determined based on historical access records. Optionally, cold data includes at least one of the following:

[0113] (1) Data in the database that has not been accessed within the first period;

[0114] In one embodiment, a first index node is selected from the last index node of the B+ tree index, and the first index node is used to index data that has not been accessed within a first time period. For example, the first index node is used to index data that has not been accessed within 30 days.

[0115] In another embodiment, a first index node is selected from all index nodes of the B-tree index, and the first index node is used to index data that has not been accessed within a first time period. For example, the first index node is used to index data that has not been accessed within 30 days.

[0116] Optionally, the first duration is a preset duration. In one embodiment, the technician can update the definition of cold data according to the real-time database access situation, that is, the technician can determine the value of the first duration according to the real-time database access situation. In another embodiment, the computing device can automatically update the value of the first duration according to the real-time database access situation. If the overall amount of data accessed to the database within a period of time is small, a larger first duration can be set to make the definition of cold data more accurate.

[0117] (2) Data in the database whose access frequency is less than the frequency threshold;

[0118] In one embodiment, in the last layer of index nodes of the B+ tree index, a first index node is selected, and the first index node is used to index data with an access frequency less than a frequency threshold. For example, the first index node is used to index data with a historical access frequency less than 5 times / day.

[0119] In another embodiment, a first index node is selected from all index nodes of the B-tree index, and the first index node is used to index data with an access frequency less than a frequency threshold. For example, the first index node is used to index data with a historical access frequency less than 5 times / day.

[0120] In the present application, the first index node is an index node that indexes data with low historical access popularity. Compressing the first index node avoids frequent decompression operations afterwards. Because the first index node indexes cold data, cold data will not be accessed frequently. At this time, it is only necessary to decompress the first index node in a few cases where the cold data is accessed.

[0121] Optionally, the frequency threshold is a pre-set threshold. In one embodiment, the technician can update the definition of cold data according to the real-time database access situation, that is, the technician can determine the value of the frequency threshold according to the real-time database access situation. In another embodiment, the computing device can automatically update the value of the frequency threshold according to the real-time database access situation. If the overall amount of data accessed to the database within a period of time is small, a smaller frequency threshold can be set to make the definition of cold data more accurate.

[0122] 2. The first index node is an index node for indexing low-reuse data in the database, where the low-reuse data refers to data shared by services that does not exceed a quantity threshold;

[0123] In one embodiment, in the last layer of index nodes of the B+ tree index, a first index node is selected, and the first index node is used to index low-reuse data in the database. For example, the first index node is used to index data used only by a single service.

[0124] In one embodiment, a first index node is selected from all index nodes of the B-tree index, and the first index node is used to index low-reuse data in the database. For example, the first index node is used to index data used only by a single service.

[0125] Optionally, the quantity threshold is a pre-set threshold. In one embodiment, the technician can update the definition of low reuse data according to the real-time business data reuse situation, that is, the technician can determine the value of the quantity threshold according to the real-time business data reuse situation. In another embodiment, the computing device can automatically update the value of the quantity threshold according to the real-time business data reuse situation. For example, if the data in the database as a whole supports fewer businesses, a smaller quantity threshold can be set to make the definition of low reuse data more accurate.

[0126] In the database, there may be some data that is shared by multiple businesses. For example, the content that a user has read in the past and the content in the user's favorites may point to the same content. If there is some data in the database that is shared by more businesses, it means that the data is more likely to be accessed again, and if there is some data in the database that is shared by fewer businesses, it means that the data is more likely to be cold data. In this application, the index node of the low-reuse data in the database is compressed and indexed to avoid frequent decompression operations later.

[0127] 3. The first index node is an index node whose expected compression rate exceeds the compression rate threshold.

[0128] In one embodiment, in the last layer of index nodes of the B+ tree index, the first index node is selected, and the first index node is an index node whose expected compression rate of internal data exceeds the compression rate threshold. For example, the first index node is an index node whose expected compression rate reaches 50%.

[0129] In another embodiment, a first index node is selected from all index nodes of the B-tree index, and the first index node is an index node whose expected compression rate of internal data exceeds a compression rate threshold. For example, the first index node is an index node whose expected compression rate reaches 50%.

[0130] Optionally, the compression rate threshold is a pre-set threshold. In one embodiment, the technician can update the compression rate threshold according to the real-time performance consumption of the computing device, that is, the technician can determine the value of the quantity threshold according to the real-time performance consumption of the computing device that performs the compression operation. In another embodiment, the computing device can automatically update the value of the compression rate threshold according to the real-time performance consumption. For example, if the real-time performance consumption of the computing device is large, a larger compression rate threshold can be set to reduce the number of compressed index nodes, thereby saving the performance of the computing device.

[0131] In the present application, the index nodes whose expected compression rate exceeds the compression rate threshold are compressed, which can improve the use effect of compression resources, and the compression resources include computing resources for performing compression operations and storage resources for storing compressed packages. In the case of limited compression resources, only the compression resources are consumed on the index nodes with better expected compression effects, and the compression resources are not wasted on the index nodes with poor expected compression effects, which can make full use of the limited compression resources and improve the use effect of the compression resources.

[0132] It should be noted that the three selection conditions introduced above can be used simultaneously, or only one of them can be used, or two of them can be selected for use. This application does not impose any restrictions on this, and these usage methods should all fall within the scope of protection of this application.

[0133] In the present application, the index nodes are scanned, and the internal data of the first index node that meets the selection condition is compressed. In one embodiment, the timing of the scan is determined based on at least one of the following methods.

[0134] A first possible determination method is to automatically scan the tree index structure once every first scanning time period according to a fixed period.

[0135] A second possible determination method: when the storage space occupied by the tree index structure exceeds the storage space threshold, the tree index structure is scanned to compress the internal data of the first index node that meets the selection condition.

[0136] Since the data stored in the database changes dynamically, the data stored in the database may gradually increase during the use of the database, and the storage space occupied by the tree index structure will gradually increase. In the present application, when the storage space occupied by the tree index structure is large, the tree index structure is streamlined and compressed, and the index nodes in the tree index structure that meet the selection conditions are compressed, which can reduce the volume of the tree index structure and save storage space.

[0137] A third possible determination method: when the number or proportion of cold data exceeds a specified value, the tree index structure is scanned to compress the internal data of the first index node indexing the cold data.

[0138] In the database, the cold data stored may gradually increase over time, which results in a large number of index nodes indexing to cold data in the tree index structure. In this application, when the number or proportion of cold data is large, the tree index structure will be streamlined and compressed, and the index nodes indexed to cold data will be compressed to reduce the volume of the tree index structure and save storage space. Optionally, the specified value changes dynamically over time, and the specified value can be selected by the technician based on actual needs, for example, the technician selects it based on the current usage of storage resources and the number of cold data in the current database.

[0139] Next, several exemplary index compression methods are introduced.

[0140] The first index compression method

[0141] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, and the compressed package is stored in the second index node. The second index node is a sibling index node of the first index node in the tree index structure. The internal data includes an index keyword and an access address of target access data. The internal data is the data stored in the first index node.

[0142] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0143] Optionally, the second index node is a sibling index node adjacent to the first index node in the tree index structure. Furthermore, the index keyword contained in the second index node is greater than the index keyword in the first index node, or the index keyword contained in the second index node is less than the index keyword in the first index node. The present application stores the compressed package in the sibling index node of the first index node, and there is no need to set up an additional storage location to store the compressed package. The method of storing the compressed package provided by the present application can avoid the overhead of additional storage space.

[0144] Figure 3 A schematic diagram of an index compression method provided by an exemplary embodiment of the present application is shown.

[0145] Figure 3Part (A) shows a two-layer tree index structure. Assume that the scan selects index node 310 in the tree index structure. There are four index keywords (K1 to K4) in index node 310, as well as access addresses (R1 to R4) of target access data corresponding to each index keyword. Each access address is used to indicate the corresponding target access data (V1 to V4).

[0146] like Figure 3 As shown in part (B), the internal data of index node 310 is compressed to obtain a compressed package, and the compressed package is stored in the storage space of index node 320. At this time, the storage space of index node 320 contains both ordinary index rows and compressed packages. Optionally, index nodes 320 and 330 are made to be adjacent sibling index nodes. For example, the page number of index node 330 is added to the storage space of index node 320 as the page number of the sibling index node, and the page number of index node 320 is added to the storage space of index node 330 as the page number of the sibling index node.

[0147] like Figure 3 As shown in part (C), the storage space of index node 310 is reclaimed at an appropriate time. Optionally, the connection between index node 310 and index node 320 and index node 330 is also disconnected. For example, the page number of index node 310 is deleted from the storage space of index node 320 and index node 330. Reclaiming the storage space of index node 310 means clearing all data in the storage space of index node 310, so that the storage space becomes a new available storage space.

[0148] In one embodiment, when querying a tree index structure, if the first index node is hit, a compressed package of the first index node is obtained from the storage space of the second index node, the compressed package of the first index node is decompressed, and the internal data of the first index node is restored. Based on the characteristics of the tree index structure, when querying the tree index structure, a layer-by-layer query will be performed according to the index keyword. If the current query needs to obtain the first index node corresponding to the first keyword interval, and the first index node has been compressed, based on the agreed storage location rule, it will be known to obtain the compressed package of the first index node from the second index node corresponding to the keyword interval (second keyword interval) adjacent to the first keyword interval. At this time, by utilizing the characteristics of the tree index structure and the agreed storage location rule, the present application can obtain the compressed package of the first index node more quickly.

[0149] In summary, the first index compression method stores the compressed package of the internal data of the first index node in the sibling index node, and the internal data of the first index node includes the index keyword and the access address of the target access data. The first index compression method is suitable for querying the data in the heap organization table. Each data stored in the heap organization table has a unique access address, represented by RID. At this time, the first index node contains at least one<Key,RID> A tuple of , where Key is the index keyword.

[0150] The second index compression method

[0151] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, the compressed package is stored in a newly created independent file, and the metadata of the compressed package is stored in a second index node, the second index node is a sibling index node of the first index node in the tree index structure, the internal data includes an index keyword and an access address of target access data, and the internal data is the data stored in the first index node.

[0152] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0153] Optionally, the second index node is a sibling index node adjacent to the first index node in the tree index structure. Furthermore, the index keyword contained in the second index node is greater than the index keyword in the first index node, or the index keyword contained in the second index node is less than the index keyword in the first index node. For the compressed compressed package, the present application only stores the metadata of the compressed package in the storage space of the second index node. The data volume of the metadata is smaller than the data volume of the entire compressed package. Compared with storing the entire compressed package, the present application stores the metadata of the compressed package in the storage space of the second index node to avoid excessive occupation of the storage space of the second index node. In addition, if the volume of the compressed package obtained after compression is large, the advantage of storing only the metadata of the compressed package in the storage space of the second index node is more obvious.

[0154] Figure 4 A schematic diagram of an index compression method provided by an exemplary embodiment of the present application is shown.

[0155] Figure 4 Part (A) shows a two-layer tree index structure. Assume that the scan selects index node 410 in the tree index structure. There are four index keywords (K1 to K4) in index node 410, as well as access addresses (R1 to R4) of target access data corresponding to each index keyword. Each access address is used to indicate the corresponding target access data (V1 to V4).

[0156] like Figure 4As shown in part (B), the internal data of index node 410 is compressed to obtain a compressed package, the meta information of the compressed package is stored in index node 420, and the compressed package is stored in an independent file. At this time, the storage space of index node 420 contains both ordinary index rows and the meta information of the compressed package. Optionally, index nodes 420 and 430 are made to be adjacent sibling index nodes to each other, for example, the page number of index node 430 is added to the storage space of index node 420 as the page number of the sibling index node, and the page number of index node 420 is added to the storage space of index node 430 as the page number of the sibling index node.

[0157] like Figure 4 As shown in part (C), the storage space of index node 410 is reclaimed at an appropriate time, and optionally, the connection between index node 410 and index node 420 and index node 430 is also disconnected, for example, the page number of index node 410 is deleted from the storage space of index node 420 and index node 430. Reclaiming the storage space of index node 410 means clearing all data in the storage space of index node 410, so that the storage space becomes a new available storage space.

[0158] In one embodiment, in the process of querying the tree index structure, if the first index node is hit, the metadata of the compressed package of the first index node is obtained from the storage space of the second index node; according to the acquisition position indicated by the metadata, the compressed package of the first index node is obtained. The compressed package of the first index node is decompressed to restore the internal data of the first index node. The present application stores the metadata of the compressed package in the sibling index nodes of the first index node, and utilizes the characteristics of the tree index structure and the agreed storage location rules to obtain the metadata of the compressed package of the first index node more quickly, thereby obtaining the compressed package more quickly.

[0159] In summary, the second index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and stores the metadata of the compressed package in the sibling index node, and the internal data of the first index node includes the index keyword and the access address of the target access data. The second index compression method is suitable for querying data in the heap organization table. Each data stored in the heap organization table has a unique access address, represented by RID. At this time, the first index node contains at least one<Key,RID> A tuple of , where Key is the index keyword.

[0160] The third index compression method

[0161] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, and the compressed package is stored in the second index node. The second index node is a sibling index node of the first index node in the tree index structure. The internal data includes index keywords and target access data. The internal data is the data stored in the first index node.

[0162] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0163] Optionally, the second index node is a sibling index node adjacent to the first index node in the tree index structure. Furthermore, the index keyword contained in the second index node is greater than the index keyword in the first index node, or the index keyword contained in the second index node is less than the index keyword in the first index node. The present application stores the compressed package in the sibling index node of the first index node, and there is no need to set up an additional storage location to store the compressed package. The method of storing the compressed package provided by the present application can avoid the overhead of additional storage space.

[0164] Figure 5 A schematic diagram of an index compression method provided by an exemplary embodiment of the present application is shown.

[0165] Figure 5 Part (A) shows a two-layer tree index structure. Assume that the scan selects index node 510 in the tree index structure. There are four index keywords (K1 to K4) in index node 510, and target access data (V1 to V4) corresponding to each index keyword.

[0166] like Figure 5 As shown in part (B), the internal data of index node 510 is compressed to obtain a compressed package, and the compressed package is stored in index node 520. At this time, the storage space of index node 520 contains both ordinary index rows and compressed packages. Optionally, index nodes 520 and 530 are made to be adjacent sibling index nodes. For example, the page number of index node 530 is added to the storage space of index node 520 as the page number of the sibling index node, and the page number of index node 520 is added to the storage space of index node 530 as the page number of the sibling index node.

[0167] like Figure 5As shown in part (C), the storage space of index node 510 is reclaimed at an appropriate time, and optionally, the connection between index node 510 and index node 520 and index node 530 is also disconnected, for example, the page number of index node 510 is deleted from the storage space of index node 520 and index node 530. Reclaiming the storage space of index node 510 means clearing all data in the storage space of index node 510, so that the storage space becomes a new available storage space.

[0168] In one embodiment, when querying the tree index structure, if the first index node is hit, the compressed package of the first index node is obtained from the storage space of the second index node, the compressed package of the first index node is decompressed, and the internal data of the first index node is restored. The present application stores the compressed package in the sibling index node of the first index node, and utilizes the characteristics of the tree index structure and the agreed storage location rules to obtain the compressed package of the first index node more quickly.

[0169] To sum up, the third index compression method will compress the internal data of the first index node and store it in the brother index node, and the internal data of the first index node includes index keywords and target access data. The third index compression method is suitable for querying data in the index organization table, and the index organization table itself is a tree index structure.

[0170] The fourth index compression method

[0171] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, the compressed package is stored in a newly created independent file, and the metadata of the compressed package is stored in a second index node, the second index node is a sibling index node of the first index node in the tree index structure, the internal data includes index keywords and target access data, and the internal data is the data stored in the first index node.

[0172] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0173] Optionally, the second index node is a sibling index node adjacent to the first index node in the tree index structure. Furthermore, the index keyword contained in the second index node is greater than the index keyword in the first index node, or the index keyword contained in the second index node is less than the index keyword in the first index node. For the compressed compressed package, the present application only stores the metadata of the compressed package in the storage space of the second index node. The data volume of the metadata is smaller than the data volume of the entire compressed package. Compared with storing the entire compressed package, the present application stores the metadata of the compressed package in the storage space of the second index node to avoid excessive occupation of the storage space of the second index node. In addition, if the volume of the compressed package obtained after compression is large, the advantage of storing only the metadata of the compressed package in the storage space of the second index node is more obvious.

[0174] Figure 6 A schematic diagram of an index compression method provided by an exemplary embodiment of the present application is shown.

[0175] Figure 6 Part (A) shows a two-layer tree index structure. Assume that the scan selects index node 610 in the tree index structure. There are four index keywords (K1 to K4) in index node 610, and target access data (V1 to V4) corresponding to each index keyword.

[0176] like Figure 6 As shown in part (B), the internal data of index node 610 is compressed to obtain a compressed package, the meta information of the compressed package is stored in index node 620, and the compressed package is stored in an independent file. At this time, the storage space of index node 620 contains both ordinary index rows and the meta information of the compressed package. Optionally, index nodes 620 and 630 are made to be adjacent sibling index nodes. For example, the page number of index node 630 is added to the storage space of index node 620 as the page number of the sibling index node, and the page number of index node 620 is added to the storage space of index node 630 as the page number of the sibling index node.

[0177] like Figure 6 As shown in part (C), the storage space of index node 610 is reclaimed at an appropriate time, and optionally, the connection between index node 610 and index node 620 and index node 630 is also disconnected, for example, the page number of index node 610 is deleted from the storage space of index node 620 and index node 630. Reclaiming the storage space of index node 610 means clearing all data in the storage space of index node 610, so that the storage space becomes a new available storage space.

[0178] In one embodiment, in the process of querying the tree index structure, if the first index node is hit, the metadata of the compressed package of the first index node is obtained from the storage space of the second index node; according to the acquisition position indicated by the metadata, the compressed package of the first index node is obtained. The compressed package of the first index node is decompressed to restore the internal data of the first index node. The present application stores the metadata of the compressed package in the sibling index nodes of the first index node, and utilizes the characteristics of the tree index structure and the agreed storage location rules to obtain the metadata of the compressed package of the first index node more quickly, thereby obtaining the compressed package more quickly.

[0179] To summarize, the fourth index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and stores the metadata of the compressed package in a brother index node, and the internal data of the first index node includes index keywords and target access data. The fourth index compression method is suitable for querying data in the index organization table, and the index organization table itself is a tree index structure.

[0180] The fifth index compression method

[0181] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, and the compressed package is stored in a newly created independent file, wherein the internal data includes an index keyword and an access address of target access data, and the internal data is the data stored in the first index node.

[0182] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0183] Figure 7 Part (A) shows a two-layer tree index structure. Assume that the scan selects index node 710 in the tree index structure. There are four index keywords (K1 to K4) in index node 710, as well as access addresses (R1 to R4) of target access data corresponding to each index keyword. Each access address is used to indicate the corresponding target access data (V1 to V4). Figure 7 In part (A), the internal data of the index node 710 is compressed to obtain a compressed package, and the compressed package is stored in an independent file.

[0184] In one embodiment, during the query of the tree index structure, if the first index node is hit, the compressed package of the first index node is obtained from the newly created independent file; the compressed package of the first index node is decompressed to restore the internal data of the first index node. When the compressed package is obtained, the present application does not need to distinguish the compressed package in the storage space shared with other data, for example, the storage location of the compressed package is determined based on the offset address in the storage space shared with other data.

[0185] In summary, the fifth index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and the internal data of the first index node includes the index keyword and the access address of the target access data. The fifth index compression method is suitable for querying the data in the heap organization table. Each data stored in the heap organization table has a unique access address, represented by RID. At this time, the first index node contains at least one<Key,RID> The fifth index compression method stores the compressed package in a newly created independent file, which can avoid occupying the storage space of other index nodes in the tree index structure. By storing the compressed package in a newly created independent file, the compressed package can be placed independently, which is conducive to the management of the compressed package.

[0186] The sixth index compression method

[0187] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, the metadata of the compressed package is stored in a newly created independent file, and the compressed package is stored in another newly created independent file, wherein the internal data of the first index node includes an index keyword and an access address of target access data, and the internal data is the data stored in the first index node.

[0188] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0189] Figure 7 Part (B) shows a two-layer tree index structure. Assume that the scan selects index node 720 in the tree index structure. There are four index keywords (K1 to K4) in index node 720, as well as an access address (R1 to R4) of the target access data corresponding to each index keyword. Each access address is used to indicate the corresponding target access data (V1 to V4). Figure 7 In part (B), the internal data of the index node 720 is compressed to obtain a compressed package, the meta information of the compressed package is stored in an independent file, and the compressed package is stored in another independent file.

[0190] In one embodiment, multiple compressed packages corresponding to the selected multiple index nodes are stored in the same file, and metadata of the multiple compressed packages corresponding to the selected multiple index nodes is stored in another file. This is conducive to hierarchical management of multiple compressed packages. When a compressed package needs to be obtained, the metadata is read from one file, and the compressed package is read from another file according to the instructions of the metadata.

[0191] In one embodiment, in the process of querying the tree index structure, if the first index node is hit, the metadata of the compressed package of the first index node is obtained from the newly created independent file, the compressed package of the first index node is obtained according to the acquisition position indicated by the metadata, the compressed package of the first index node is decompressed, and the internal data of the first index node is restored. The present application implements hierarchical query when querying the compressed package, which is not prone to errors. The present application will obtain the metadata of the compressed package from the newly created independent file, and further obtain the compressed package based on the acquisition position indicated by the metadata, thereby implementing hierarchical query of the compressed package.

[0192] In summary, the sixth index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and stores the metadata of the compressed package in another newly created independent file, and the internal data of the first index node includes the index keyword and the access address of the target access data. The sixth index compression method is suitable for querying the data in the heap organization table. Each data stored in the heap organization table has a unique access address represented by RID. At this time, the first index node contains at least one<Key,RID> The sixth index compression method stores the compressed package and its meta information in a newly created independent file, which can avoid occupying the storage space of other index nodes in the tree index structure. By storing the compressed package in a newly created independent file, the compressed package can be placed independently, which is conducive to the management of the compressed package.

[0193] The seventh index compression method

[0194] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, and the compressed package is stored in a newly created independent file, wherein the internal data of the first index node includes an index keyword and target access data, and the internal data is the data stored in the first index node.

[0195] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0196] Figure 7 Part (C) shows a two-layer tree index structure. Assume that the scan selects index node 730 in the tree index structure. There are four index keywords (K1 to K4) in index node 730, and target access data (V1 to V4) corresponding to each index keyword. Figure 7 In part (C), the internal data of the index node 730 is compressed to obtain a compressed package, and the compressed package is stored in an independent file.

[0197] In one embodiment, during the query of the tree index structure, if the first index node is hit, the compressed package of the first index node is obtained from the newly created independent file, the compressed package of the first index node is decompressed, and the internal data of the first index node is restored. When the compressed package is obtained, the present application does not need to distinguish the compressed package in the storage space shared with other data, for example, the storage location of the compressed package is determined based on the offset address in the storage space shared with other data.

[0198] In summary, the seventh index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and the internal data of the first index node includes index keywords and target access data. The seventh index compression method is suitable for querying data in the index organization table, and the index organization table itself is a tree-like index structure. In addition, the seventh index compression method stores the compressed package in a newly created independent file, which can avoid occupying the storage space of other index nodes in the tree-like index structure. By storing the compressed package in a newly created independent file, the compressed package can be placed independently, which is conducive to the management of the compressed package.

[0199] The eighth index compression method

[0200] In one embodiment, the internal data of the first index node is compressed to obtain a compressed package, the metadata of the compressed package is stored in a newly created independent file, and the compressed package is stored in another newly created independent file, wherein the internal data of the first index node includes index keywords and target access data, the internal data is the data stored in the first index node, and the internal data is the data stored in the first index node.

[0201] The index keyword is used to indicate the target index node when querying the tree index structure, and the target access data refers to the data accessed when querying the tree index structure.

[0202] Figure 7 Part (D) shows a two-layer tree index structure. Assume that the scan selects index node 740 in the tree index structure. There are four index keywords (K1 to K4) in index node 740, and target access data (V1 to V4) corresponding to each index keyword. Figure 7 In part (D), the internal data of the index node 740 is compressed to obtain a compressed package, the meta information of the compressed package is stored in an independent file, and the compressed package is stored in another independent file.

[0203] In one embodiment, multiple compressed packages corresponding to the selected multiple index nodes are stored in the same file, and metadata of the multiple compressed packages corresponding to the selected multiple index nodes is stored in another file. This is conducive to hierarchical management of multiple compressed packages. When a compressed package needs to be obtained, the metadata is read from one file, and the compressed package is read from another file according to the instructions of the metadata.

[0204] In one embodiment, in the process of querying the tree index structure, if the first index node is hit, the metadata of the compressed package of the first index node is obtained from the newly created independent file, the compressed package of the first index node is obtained according to the acquisition position indicated by the metadata, the compressed package of the first index node is decompressed, and the internal data of the first index node is restored. The present application implements hierarchical query when querying the compressed package, which is not prone to errors. The present application will obtain the metadata of the compressed package from the newly created independent file, and further obtain the compressed package based on the acquisition position indicated by the metadata, thereby implementing hierarchical query of the compressed package.

[0205] In summary, the eighth index compression method stores the compressed package of the internal data of the first index node in a newly created independent file, and stores the metadata of the compressed package in another newly created independent file, and the internal data of the first index node includes index keywords and target access data, and the eighth index compression method is suitable for querying data in the index organization table, and the index organization table itself is a tree-like index structure. In addition, the eighth index compression method stores the compressed package and the metadata of the compressed package in a newly created independent file, which can avoid occupying the storage space of other index nodes in the tree-like index structure, and the compressed package can be placed independently by storing the compressed package in a newly created independent file, which is conducive to the management of the compressed package.

[0206] It should be noted that the several index compression methods introduced above can be understood as compressing the index nodes of the last layer of the B+ tree index, and can also be understood as compressing the index nodes of the last layer of the B-tree index. That is, the above index compression methods are applicable to both B+ tree indexes and B-tree indexes.

[0207] based on Figure 2 In the optional embodiment shown, step 260 also includes Figure 8 Step 280 is shown.

[0208] Step 280: reclaim the storage space corresponding to the first index node.

[0209] Due to the existence of parallel transactions, after the internal data of the first index node is compressed, there may be other transactions that will access the first index node, so the first index node cannot be reclaimed immediately. In an embodiment of the present application, after the compression is completed, a recycle number will be assigned to the first index node, and the recycle number of the first index node is the next number of the assigned transaction number. For example, after the compression is completed, the transaction number of the current global transaction reaches 10000, and the recycle number 10001 is assigned to the first index node.

[0210] Scan the index nodes in at least two layers of index nodes again. When the first index node is scanned, if the recycling number of the first index node is less than the minimum number of the current active transaction, recycle the storage space corresponding to the first index node. For example, when the first index node is scanned, the minimum number of the current active transaction has reached 15000, and the recycling number of the first index node is 10001. Therefore, it is determined that no more transactions will access the first index node in the future, and the storage space corresponding to the first index node will be reclaimed at this time. Optionally, the storage space of the first index node is 8KB or 16KB.

[0211] Optionally, the scanning order of the second scan can refer to the relevant introduction of the reference order of selecting the first index node in the first scan above, which will not be repeated here. Optionally, the second scan can be a scan performed after the first scan above is completed, or it can be a scan performed in parallel with the first scan, as long as the scanning progress of the second scan is slower than that of the first scan.

[0212] In summary, the index compression method proposed in this application can actively recycle the storage space of the index node after the index node is compressed, improve the utilization rate of the index node, and reduce page expansion in the business process. In addition, this application takes into account the situation that the storage space of the first index node cannot be reclaimed immediately after compression. After compression, a latest transaction number is assigned as the recycling number. When recycling, the size relationship between the recycling number and the current smallest active transaction number will be judged, which can ensure that no other transactions access the first index node after recycling, which not only realizes the active recycling of the storage space of the index node, but also ensures the normal execution of other transactions.

[0213] The present application also provides an index compression device. The index compression device can be implemented as part or all of a computer device by software, hardware, or a combination of both. Fig. 9 The device includes: an acquisition module 901, a scanning module 902 and a compression module 903.

[0214] An acquisition module 901 is used to acquire a tree index structure in a database, where the tree index structure includes at least two layers of index nodes;

[0215] A scanning module 902, configured to scan index nodes in at least two layers of index nodes;

[0216] The compression module 903 is used to compress the first index node to obtain a compressed package when the scanned first index node meets the selection condition.

[0217] In an optional embodiment, the first index node satisfies the selection condition of the index node, including at least one of the following:

[0218] The first index node is an index node for indexing cold data in the database, where cold data refers to data whose access heat is less than a heat threshold;

[0219] The first index node is an index node for indexing low-reuse data in the database, where the low-reuse data refers to data shared by services that does not exceed a quantity threshold;

[0220] An index node whose expected compression ratio exceeds a compression ratio threshold value.

[0221] In an optional embodiment, the cold data includes at least one of the following:

[0222] Data in the database that has not been accessed within the first period;

[0223] Data in the database whose access frequency is less than the frequency threshold.

[0224] In an optional embodiment, the device further includes a storage module 904 .

[0225] A storage module 904, configured to store the compressed package in a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure;

[0226] or,

[0227] The storage module 904 is used to store the meta information of the compressed package in the storage space of the second index node.

[0228] In an optional embodiment, the storage module 904 is further used to store the compressed package in a newly created independent file;

[0229] or,

[0230] Store the metadata of the compressed package in a newly created independent file.

[0231] In an optional embodiment, the device further includes a recycling module 905, and the recycling module 905 is used to recycle the storage space corresponding to the first index node.

[0232] In an optional embodiment, the recycling module 905 is further configured to use the next number of the allocated transaction number as the recycling number of the first index node;

[0233] Scan the index nodes in at least two levels of index nodes again;

[0234] When the first index node is scanned, if the recycling number of the first index node is smaller than the minimum number of the currently active transaction, the storage space corresponding to the first index node is recycled.

[0235] In an optional embodiment, the tree index structure is a B+ tree index, and the scanning module 902 is also used to scan the index nodes one by one in the last layer of index nodes of at least two layers of index nodes according to a reference order, and the reference order includes the order of index keywords existing in the index nodes from large to small, or from small to large.

[0236] In an optional embodiment, the data stored in the first index node includes any one of the following:

[0237] Index keywords and target access data, where target access data refers to the data accessed when querying the tree index structure;

[0238] Index keyword and access address of target access data.

[0239] In an optional embodiment, the device also includes a query module 906; the query module 906 is used to obtain a compressed package of the first index node if the first index node is hit during the query of the tree index structure; decompress the compressed package to restore the internal data of the first index node.

[0240] In an optional embodiment, the query module 906 is further configured to obtain a compressed package of the first index node from a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure.

[0241] In an optional embodiment, the query module 906 is further configured to obtain meta information of the compressed package of the first index node from the storage space of the second index node; and obtain the compressed package of the first index node according to the acquisition position indicated by the meta information.

[0242] In an optional embodiment, the query module 906 is further configured to obtain a compressed package of the first index node from the newly created independent file.

[0243] In an optional embodiment, the query module 906 is further configured to obtain meta information of the compressed package of the first index node from the newly created independent file; and obtain the compressed package of the first index node according to the acquisition position indicated by the meta information.

[0244] In summary, the present application will select the first index node in the tree index structure according to the selection conditions, compress the first index node to obtain a compressed package, and compressing the first index node can improve the overall compression rate of the database and reduce the size of the database. Moreover, by selective compression rather than unselectively compressing multiple index nodes in full, the compression effect is guaranteed and the compression of meaningless index nodes is avoided. Moreover, the index compression method provided by the present application can be applied to any tree index structure, for example, it can be applied to both B-tree index and B+ tree index.

[0245] Among them, the acquisition module 901, the scanning module 902, the compression module 903, the storage module 904, the recovery module 905 and the query module 906 can all be implemented by software, or can be implemented by hardware. Exemplarily, the acquisition module 901 is taken as an example to introduce the implementation of the acquisition module 901. Similarly, the implementation of the scanning module 902, the compression module 903, the storage module 904, the recovery module 905 and the query module 906 can refer to the implementation of the A module.

[0246] As an example of a software functional unit, the acquisition module 901 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 901 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0247] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0248] As an example of a hardware functional unit, the acquisition module 901 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 901 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0249] The multiple computing devices included in the acquisition module 901 can be distributed in the same region or in different regions. The multiple computing devices included in the A module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 901 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0250] It should be noted that, in other embodiments, the acquisition module 901 can be used to execute any step in the index compression method, the scanning module 902 can be used to execute any step in the index compression method, the compression module 903 can be used to execute any step in the index compression method, the storage module 904 can be used to execute any step in the index compression method, the recycling module 905 can be used to execute any step in the index compression method, and the query module 906 can be used to execute any step in the index compression method. The steps that the acquisition module 901, the scanning module 902, the compression module 903, the storage module 904, the recycling module 905, and the query module 906 are responsible for implementing can be specified as needed, and the acquisition module 901, the scanning module 902, the compression module 903, the storage module 904, the recycling module 905, and the query module 906 respectively implement different steps in the index compression method to achieve the full functions of the index compression device.

[0251] It should be noted that: the index compression device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when compressing the index. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the index compression device provided in the above embodiment and the index compression method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0252] The present application also provides a computing device, please refer to Fig.10 , the computing device includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other through the bus 1002. The computing device 1000 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1000.

[0253] The bus 1002 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (eg, the memory 1006, the processor 1004, and the communication interface 1008).

[0254] The processor 1004 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0255] The memory 1006 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0256] The memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to respectively implement the functions of the acquisition module, scanning module, compression module, storage module and recycling module, thereby implementing the index compression method. That is, the memory 1006 stores instructions for executing the index compression method.

[0257] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or a communication network.

[0258] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0259] like Fig.11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the index compression method. Similar to the computing device 1000, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002.

[0260] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the index compression method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the index compression method.

[0261] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the index compression device. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more modules of the acquisition module, the scanning module, the compression module, the storage module and the recycling module.

[0262] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.12 A possible implementation is shown. Fig.12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1006 in the computing device 1000A stores instructions for executing the functions of the acquisition module, the scanning module, and the compression module. At the same time, the memory 1006 in the computing device 1000B stores instructions for executing the functions of the storage module, the recycling module, and the query module.

[0263] Fig.12 The connection method between the computing device clusters shown can be based on the consideration that the index compression method provided in the present application needs to perform a large number of storage data operations, storage space recovery operations and query operations, so it is considered that the functions implemented by the storage module, recovery module and query module are handed over to the computing device 1000B for execution.

[0264] It should be understood that Fig.12 The functions of the computing device 1000A shown in FIG. 1000A may also be completed by multiple computing devices 1000B. Similarly, the functions of the computing device 1200B may also be completed by multiple computing devices 1000A.

[0265] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.11 and Fig.12 The connection mode of the computing device cluster is different in that the memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the index compression method.

[0266] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the index compression method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the index compression method.

[0267] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the index compression method.

[0268] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the index compression method.

[0269] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0270] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0271] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0272] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An index compression method, characterized in that: The method comprises: Acquire a tree index structure in a database, wherein the tree index structure includes at least two layers of index nodes; Scanning the index nodes in the at least two layers of index nodes; When the scanned first index node meets the selection condition, the first index node is compressed to obtain a compressed package.

2. The method according to claim 1, characterized in that The first index node satisfies a selection condition, including at least one of the following: The first index node is an index node for indexing cold data in the database, where the cold data refers to data whose access heat is less than a heat threshold; The first index node is an index node for indexing low-multiplexed data in the database, wherein the low-multiplexed data refers to data shared by services that does not exceed a quantity threshold; The first index node is an index node whose expected compression rate exceeds a compression rate threshold.

3. The method according to claim 2, characterized in that The cold data includes at least one of the following data: Data in the database that has not been accessed within a first period of time; The data in the database has an access frequency less than a frequency threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: storing the compressed package in a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure; or, The meta information of the compressed package is stored in the storage space of the second index node.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Storing the compressed package in a newly created independent file; or, The meta information of the compressed package is stored in the newly created independent file.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Reclaim the storage space corresponding to the first index node.

7. The method according to claim 6, characterized in that After compressing the first index node to obtain a compressed package, the method further includes: Using the next number of the allocated transaction number as the recycling number of the first index node; The reclaiming the storage space corresponding to the first index node includes: Scanning the index nodes in the at least two layers of index nodes again; When the first index node is scanned, if the recycling number of the first index node is smaller than the minimum number of the currently active transaction, the storage space corresponding to the first index node is recycled.

8. The method according to any one of claims 1 to 7, characterized in that: The tree-like index structure is a B+ tree index, and scanning the index nodes in the at least two layers of index nodes includes: In the last layer of index nodes of the at least two layers of index nodes, the index nodes are scanned according to a reference order, where the reference order includes an order of index keywords existing in the index nodes from large to small, or from small to large.

9. The method according to any one of claims 1 to 8, characterized in that: The first index node stores any one of the following data: Index keywords and target access data, wherein the target access data refers to data accessed when querying the tree index structure; An index keyword and an access address of the target access data.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: In the process of querying the tree index structure, if the first index node is hit, obtaining a compressed package of the first index node; Decompress the compressed package to restore the internal data of the first index node.

11. The method according to claim 10, characterized in that The obtaining of the compressed package of the first index node includes: Acquire a compressed package of the first index node from a storage space of a second index node, where the second index node is a sibling index node of the first index node in the tree index structure; or, The meta information of the compressed package of the first index node is obtained from the storage space of the second index node; and the compressed package of the first index node is obtained according to the obtaining position indicated by the meta information.

12. The method according to claim 10, characterized in that The obtaining of the compressed package of the first index node includes: Obtaining a compressed package of the first index node from the newly created independent file; or, The meta information of the compressed package of the first index node is obtained from the newly created independent file; and the compressed package of the first index node is obtained according to the obtaining position indicated by the meta information.

13. An index compression device, characterized in that: The device comprises: An acquisition module, used to acquire a tree index structure in a database, wherein the tree index structure includes at least two layers of index nodes; A scanning module, used for scanning the index nodes in the at least two layers of index nodes; The compression module is used to compress the first index node scanned to obtain a compressed package when the first index node meets the selection condition.

14. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 12.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 12.

Images