An evolutionary method, system, and device for an evolving database index structure.

Abstract

This application provides a method, system, and device for evolving an evolvable database index structure, belonging to the field of database technology. The method includes dividing a data table into multiple data segments; determining the access characteristics and system resource status of each data segment; determining the access characteristic similarity between adjacent data segments based on the access characteristics and system resource status; and dynamically splitting or merging the data segments based on the access characteristic similarity. Based on the access characteristics and a preset index meta-model, it is determined whether each data segment meets preset evolution trigger conditions. If the preset evolution trigger conditions are met, the target index type corresponding to the data segment is determined, and the data segment is converted from the current index structure to the target index structure corresponding to the target index type through incremental migration. This method achieves adaptive evolution of the database index structure.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to an evolution method, system, and device for an evolving database index structure. Background Technology

[0002] Database index structures play a crucial role in query performance, write performance, storage overhead, and adaptability. Existing database systems typically offer various types of index structures, such as B-trees, Log-Structured Merge-Tree (LSM-Tree), and bitmap indexes. Each index structure has its own performance advantages under specific data distribution and access patterns.

[0003] However, in real-world business scenarios, data distribution and access patterns often change dynamically over time, such as the migration of hot and cold data, seasonal fluctuations in user behavior, and periodic switching of read and write loads. The aforementioned index structures are optimized for specific scenarios, and this static design struggles to adapt to dynamically changing business loads and data distributions.

[0004] Furthermore, when access patterns change significantly, such as during promotional activities or sudden surges in access causing indexes to repeatedly switch between different structures, administrators need to manually adjust the index structure, resulting in high labor costs, significant risks, and long delays. Meanwhile, industry-standard solutions such as multi-index coexistence and incremental index building either incur high maintenance costs or only optimize the internal structure of a single index, failing to address the issue of index mismatch with dynamic load. Summary of the Invention

[0005] This application provides a method, system, and device for evolving an evolving database index structure, which addresses the technical problems of current database index structures being static, having high maintenance costs, and being unable to adaptively evolve.

[0006] In a first aspect, embodiments of this application provide an evolutionary method for an evolving database index structure, the method comprising:

[0007] Divide the data table into multiple data segments;

[0008] Determine the access characteristics and system resource status of each data segment;

[0009] Based on the access characteristics and the system resource status, the access characteristic similarity between adjacent data segments is determined, and the data segments are dynamically split or merged based on the access characteristic similarity.

[0010] Based on the access characteristics and the preset index meta-model, determine whether each data segment meets the preset evolution triggering condition;

[0011] If the preset evolution triggering condition is met, the target index type corresponding to the data segment is determined, and the data segment is converted from the current index structure to the target index structure corresponding to the target index type through incremental migration.

[0012] In one implementation of this application, the step of dynamically splitting or merging the data fragments based on the access feature similarity specifically includes:

[0013] When the similarity of the access features is continuously less than a preset splitting threshold within N consecutive statistical windows, the data segment is split; where N is a natural number greater than 1.

[0014] When the access feature similarity is continuously greater than the preset merging threshold within N consecutive statistical windows, the data segment is merged; wherein, the preset splitting threshold is less than the preset merging threshold; the splitting and merging are set to be performed when the system resource utilization rate is less than the preset resource threshold, and the minimum data size of the split data segment is not less than the first preset threshold, and the maximum data size of the merged data segment is not greater than the second preset threshold.

[0015] In one implementation of this application, determining the access characteristics and system resource status of each data segment specifically includes:

[0016] Coarse-grained statistics are performed on all the data segments at a first preset period, and a first preset index is collected;

[0017] Based on the first preset index, the corresponding hot data segments are selected, and the hot data segments are sampled in a fine-grained manner at a second preset period to collect the second preset index; the second preset index is a fine-grained index that is different from the first preset index.

[0018] When the system resource utilization rate exceeds the preset high load threshold, the fine-grained sampling is paused.

[0019] In one implementation of this application, the index metamodel is constructed by abstracting the capability cost information of various heterogeneous index structures, and the index metamodel includes at least:

[0020] Index capability tags are used to identify the suitability of an index structure for point queries, range scans, sequential write optimizations, or bitmap optimizations.

[0021] Evolutionary capability label, used to indicate the target index type that the index structure allows for transformation;

[0022] In addition, storage overhead estimates and build time estimates.

[0023] In one implementation of this application, the determination that the preset evolution triggering condition is met specifically includes:

[0024] A linear fit is performed on the access index of the data segment, and the goodness-of-fit value is calculated.

[0025] When the goodness-of-fit value is greater than the preset trend threshold, the change in the access index is determined to be a trend feature; otherwise, it is determined to be a temporary fluctuation and the determination is terminated.

[0026] After determining the trend feature, when the popularity score of the data segment is greater than the preset popularity threshold, the system resource utilization rate is less than the preset safety threshold, and the data segment is not within the preset evolution cooling period, the evolution triggering condition is determined to be met.

[0027] In one implementation of this application, the duration of the preset evolution cooldown period is dynamically calculated based on the construction cost and expected performance gains of this evolution, and the specific calculation formula is as follows:

[0028] ;

[0029] in, To preset the duration of the evolutionary cooldown period, Based on the cooldown time, To build cost, To achieve the expected performance gains, the duration of the preset evolution cooling period is set with a preset upper limit and a preset lower limit.

[0030] The method further includes:

[0031] When the heat score of the data segment exceeds the preset super hot spot threshold, the preset evolution cooling period is forcibly terminated.

[0032] In one implementation of this application, the determination that the preset evolution triggering condition is met further includes:

[0033] The evolution cost is calculated based on the construction cost, storage overhead, expected performance gains, and the historical average net loss of the data segment in this evolution.

[0034] The formula for calculating the evolutionary cost is as follows:

[0035] ;

[0036] in, For the evolutionary cost, For index build time, The index occupies storage space. For the expected performance gains of this evolution, The average net loss of the data segment across several historical evolutions is given, and , The cost of the nth historical evolution, For the actual benefit of the nth historical evolution, For the number of historical evolutions, , , These are preset weight coefficients, which correspond to the importance weights of construction time, storage overhead, and net revenue, respectively.

[0037] When the evolution cost is less than a preset cost threshold, the evolution operation is triggered.

[0038] In one implementation of this application, the step of incrementally migrating the data fragment from the current index structure to the target index structure corresponding to the target index type specifically includes:

[0039] Create the target index structure;

[0040] Using index nodes as the smallest unit, and following the order determined by node access frequency and key value range continuity, the index nodes are incrementally migrated from the current index structure to the target index structure in batches.

[0041] During the migration process, a double write operation is performed on the write operation corresponding to the index node being migrated, simultaneously writing to both the current index structure and the target index structure;

[0042] After all the index nodes have been migrated and their consistency has been verified, the current index pointer of the data segment is switched from the current index structure to the target index structure through an atomic operation, so that the target index structure becomes effective.

[0043] If the reference count of the current index structure is detected to be zero, the storage resources occupied by the current index structure are released.

[0044] Secondly, embodiments of this application also provide an evolutionary system for an evolving database index structure, the system comprising:

[0045] The partitioning module is used to divide a data table into multiple data segments;

[0046] The first determining module determines the access characteristics and system resource status of each data segment;

[0047] The execution module determines the access feature similarity between adjacent data segments based on the access features and the system resource status, and performs dynamic splitting or merging of the data segments based on the access feature similarity.

[0048] The second determining module is used to determine, based on the access characteristics and the preset index meta-model, whether each of the data segments meets the preset evolution triggering conditions.

[0049] An evolution module is used to determine the target index type corresponding to the data fragment when the evolution triggering condition is met, and to convert the data fragment from the current index structure to the target index structure corresponding to the target index type through incremental migration.

[0050] Thirdly, embodiments of this application also provide an evolution device for an evolving database index structure, the device comprising:

[0051] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described method for evolving an evolving database index structure.

[0052] Compared with the prior art, the significant advantages of this application are as follows:

[0053] Through the above-described scheme, this application provides an evolutionary database index structure solution that enables evolution. It divides the data table into multiple data segments and manages the index independently for each segment, achieving fine-grained optimization of the index structure. Furthermore, it dynamically partitions or merges data segments based on the similarity of access features corresponding to the data access model, allowing the data segments corresponding to the index structure to change dynamically and simultaneously enhancing the dynamism of the corresponding index structure. Simultaneously, it utilizes an index meta-model to implement conversion decisions for different index structures and uses incremental migration to achieve smooth online evolution of the index structure. This allows the entire evolution process to proceed without downtime or blocking read / write operations, achieving seamless index structure switching without manual intervention. This solves the technical problems of current database index structures being static, having high maintenance costs, and being unable to adaptively evolve their index structures. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart illustrating an evolution method for an evolving database index structure, as described in an embodiment of this application.

[0056] Figure 2 This is a schematic diagram of the structure of an evolutionary system of a database index structure capable of evolution, as described in an embodiment of this application.

[0057] Figure 3 This is a schematic diagram of the structure of an evolution device for an evolving database index structure, as described in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] In real-world database application scenarios, data distribution and access patterns often change dynamically over time, such as cold / hot data migration, seasonal fluctuations in user behavior, and periodic switching of read / write loads. These index structures are optimized for specific scenarios, and such static designs are ill-suited to adapt to dynamically changing business loads and data distributions.

[0060] Furthermore, when access patterns change significantly, such as during promotional activities or sudden surges in access causing indexes to repeatedly switch between different structures, administrators need to manually adjust the index structure, resulting in high labor costs, significant risks, and long delays. Meanwhile, industry-standard solutions such as multi-index coexistence and incremental index building either incur high maintenance costs or only optimize the internal structure of a single index, failing to address the issue of index mismatch with dynamic load.

[0061] Based on this, embodiments of this application provide an evolution method, system, and device for an evolving database index structure, which solves the technical problems of current database index structures being static, having high maintenance costs, and being unable to adaptively evolve the index structure.

[0062] The various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0063] This application provides an evolutionary method for an evolving database index structure, such as... Figure 1 As shown, the method may include steps S101-S105:

[0064] S101 divides the data table into multiple data segments.

[0065] In other words, this application uses data segments, rather than the entire table, as the smallest control unit for index evolution. Specifically, during initialization phases such as system startup, the initial creation of an index for a table, or after a table structure change, the data table can be initially divided into multiple data segments according to preset partitioning rules. Subsequent data segment partitioning is then performed according to the following steps S102-S103, with the initial partitioning rules as follows:

[0066] The key value range of the primary key or clustered index is divided into X intervals, based on the minimum and maximum values ​​of the primary key, either evenly or according to the data distribution. Each interval serves as an initial Segment. For example, if the primary key range is 1 to 1,000,000, and the initial number of Segments is set to 100, then each Segment covers 10,000 primary key value ranges. The system creates a Segment object for each interval and records its key value range.

[0067] The data is divided according to the time interval of the time series field. Specifically, it is divided according to the minimum and maximum timestamp fields, and according to a preset fixed time granularity (such as days or hours). Each time interval is an initial segment. For example, the data from January 1, 2024 to January 31, 2024 is divided into 31 segments based on days.

[0068] The data is divided according to either physical page or data block ranges, specifically based on the table's physical storage structure. Each initial segment consists of M data pages or m data blocks, reusing existing storage boundary information from the database. Here, X, M, and m are all natural numbers. For example, each segment might consist of 100 data pages or 64MB data blocks. The system retrieves the metadata of the data pages or data blocks from the storage engine and creates segment objects sequentially in physical order.

[0069] At the same time, a default index type (such as B-tree) can be uniformly assigned to all segments during initialization, and then gradually evolved according to access characteristics.

[0070] It should be noted that the executing entity of this application may be a server. The server, as the executing entity of the evolution method of the database index structure that can evolve, is only an example. The executing entity is not limited to the server, and this application does not make any specific limitation on it.

[0071] S102, determine the access characteristics and system resource status of each data segment.

[0072] In this embodiment of the application, determining the access characteristics and system resource status of each data segment specifically includes:

[0073] Coarse-grained statistics are performed on all data segments at a first preset period, collecting a first preset indicator. Based on the first preset indicator, relevant hot data segments are selected, and fine-grained sampling is performed on these hot data segments at a second preset period, collecting a second preset indicator. The second preset indicator is a fine-grained indicator different from the first preset indicator. Fine-grained sampling is paused when the system resource utilization rate exceeds a preset high-load threshold.

[0074] Specifically, this application adopts a two-layer lightweight sampling and monitoring mechanism. The first layer is coarse-grained full statistics: coarse-grained full statistics are performed on all data segments at a first preset period (e.g., 5 seconds) to collect first preset indicators. The first preset indicators include at least: read count, write count, last update time, and data volume. To reduce statistical overhead, the system uses a probabilistic data structure for statistics: Count-Min Sketch is used to count the access frequency of each segment; HyperLogLog is used to estimate the cardinality of each segment. The core indicator collection overhead for each segment is controlled at the microsecond level. The second layer is fine-grained sampling: high-popularity data segments are selected based on the first preset indicators. Specifically, the system calculates the popularity score for each segment and selects the segments with the highest popularity scores (e.g., the top 30%) as hot data segments. The preset ratio is based on expert experience and is not specifically limited here. The popularity score calculation formula is as follows:

[0075] ;

[0076] in, Rate it based on popularity; , These are the first preset heat index weighting coefficient and the second preset heat index weighting coefficient, used to balance the contribution of read and write heat indexes, such as: , It is adapted to read-heavy, write-light scenarios by default; for write-heavy, read-light scenarios, it can be adapted to... Adjust to 0.3. Adjust it to 0.7, but the specific adjustment should be made according to the actual usage scenario; no specific limit is given here. Indicates the number of reads; This indicates the total number of reads of the entire table; Indicates the number of times it was written; This indicates the total number of writes to the entire table; Indicates the current time; Indicates the most recent update time of the data segment; This represents the time decay factor, used for historical access decay, such as... .

[0077] For the selected hot data segments, fine-grained and precise sampling is performed at a second preset period (e.g., 1 second) to collect second preset indicators. The second preset indicators include: point query ratio, range query ratio, analysis query ratio, write rate, and data distribution order.

[0078] In addition, the server can monitor the Central Processing Unit (CPU) utilization and input / output (I / O) utilization in real time. When the system resource utilization exceeds a preset high-load threshold (e.g., 80%), the second-level fine-grained sampling is automatically paused, retaining only the first-level coarse-grained statistics. The second-level sampling resumes once the system resource utilization falls below a preset safety threshold (e.g., 70%). The aforementioned access characteristics include at least the first and second preset indicators, and the system resource status includes at least CPU utilization and I / O utilization.

[0079] S103, based on access characteristics and system resource status, determine the access characteristic similarity between adjacent data segments, and perform dynamic splitting or merging of data segments based on access characteristic similarity.

[0080] This application can independently calculate the access feature similarity between adjacent data segments for each data segment, and perform splitting or merging on the data segment when the preset conditions are continuously met within multiple consecutive statistical periods based on the similarity. The formula for calculating the access feature similarity is as follows:

[0081] ;

[0082] in, For data fragments and data fragments Similarity of access features; For data fragments The corresponding eigenvector of the th Feature values ​​of each access feature dimension; For data fragments The Middle The feature values ​​of each access feature dimension; the access feature dimension includes at least four core dimensions: point query ratio, range query ratio, read-write ratio, and popularity score; This represents the total number of access feature dimensions. The feature values ​​of these dimensions can be extracted using pre-defined rules within the system, through a pre-trained neural network, or obtained through other means; this application does not impose specific limitations on this. Specifically, the percentage of point queries (equal-value queries, such as WHERE id = 123) within the statistical window can be determined; the percentage of range queries (such as WHERE id BETWEEN 1 AND 100) within the statistical window can be determined; the read ratio is the ratio of read operations to write operations within the statistical window, reflecting whether a segment is read-intensive or write-intensive; the popularity score can be directly obtained through the above embodiments. After obtaining the values ​​corresponding to each dimension, the four dimension values ​​are combined as feature values ​​to form a feature vector for calculating access feature similarity.

[0083] In this embodiment of the application, data segments are dynamically split or merged based on access feature similarity, specifically including:

[0084] When the access feature similarity is consistently less than a preset splitting threshold within N consecutive statistical windows, the data segment is split. N is a natural number greater than 1. When the access feature similarity is consistently greater than a preset merging threshold within N consecutive statistical windows, the data segment is merged. The preset splitting threshold is less than the preset merging threshold. Splitting and merging are configured to be performed when system resource utilization is less than a preset resource threshold, and the minimum data size of the split data segment is not less than a first preset threshold, and the maximum data size of the merged data segment is not greater than a second preset threshold.

[0085] In other words, this application pre-sets a statistical window, which specifically corresponds to a preset duration. This preset duration can be set based on the actual usage scenario and is not specifically limited here. The server compares the similarity of access features within N consecutive statistical windows with preset splitting thresholds and preset merging thresholds, and determines whether to split or merge the corresponding data segments based on the conditions corresponding to the splitting and merging processes. Simultaneously, before executing the splitting and merging processes, it also determines whether the system resource utilization rate is less than a preset resource threshold. If the system resource utilization rate is relatively low, less than the preset resource threshold, and the minimum data size of the split data segments is not less than the first preset threshold, then the splitting process can be executed. If the system resource utilization rate is less than the preset resource threshold and the maximum data size of the merged data segments is not greater than the second preset threshold, then the merging process can be executed. If the system resource utilization rate is greater than or equal to the preset resource threshold, then splitting and merging processes are not allowed. Furthermore, splitting and merging must also meet the data size requirements for the data segments.

[0086] The aforementioned preset splitting threshold, preset merging threshold, preset resource threshold, first preset threshold, and second preset threshold are set by users based on expert experience during actual use, and this application does not impose specific limitations on them.

[0087] The aforementioned data segmentation method is behavior-oriented and can dynamically adjust data segments based on access behavior. Compared to data segmentation based on physical location or fixed range, this method is more flexible. By adding time constraints through statistical windows and setting resource thresholds for system resource utilization, while also imposing granular restrictions on data volume, it effectively avoids the fragmentation oscillation phenomenon caused by frequent and repeated switching between splitting and merging states due to short-term fluctuations in access characteristics, which leads to wasted system resources, performance jitter, and unstable metadata.

[0088] S104, based on access characteristics and a preset index meta-model, determines whether each data segment meets the preset evolution triggering conditions.

[0089] Different data segments can correspond to different target index types at the same time.

[0090] In this embodiment of the application, the above-mentioned index meta-model is constructed by abstracting the capability cost information of various heterogeneous index structures, and the index meta-model includes at least:

[0091] The index capability label identifies the suitability of the index structure for point queries, range scans, sequential write optimizations, or bitmap optimizations. The evolution capability label indicates the target index type for which the index structure is allowed to be transformed. Additionally, storage overhead estimates and build time estimates are provided.

[0092] When constructing the index metamodel, for different types of index structures, their internal nodes are first abstracted into a unified index node structure. Each index node contains at least the following attributes: key value range (consisting of the minimum and maximum keys), a list of child node pointers, the number of records within the node, and the node access frequency.

[0093] The index metamodel module configures a pre-trained adapter for each index type. This adapter pre-sets the following transformation generation rules to convert the native index structure into a unified list of index nodes. Specifically:

[0094] For B-tree indexes, the adapter performs a depth-first traversal of all leaf nodes of the B+ tree, extracting the key value range and record count for each leaf node to generate the corresponding index node. Leaf nodes have no child nodes, therefore the list of child node pointers is empty.

[0095] For LSM-Tree indexes, the adapter treats each SSTable as an index node. The key value range of this node is the minimum and maximum key of the SSTable. The child node pointers point to references of the lower-level SSTable, and the number of records in the node is the number of records in the SSTable.

[0096] For BRIN indexes, the adapter treats each block range summary as an index node, with the key value range taking the minimum and maximum values ​​of the fields within that block range, and no child node pointers.

[0097] For bitmap indexing, the adapter treats each bitmap segment as an index node, with the key value range being the set of key values ​​corresponding to that bitmap segment, and no child node pointers.

[0098] For learned indexes, the adapter treats each model segment as an index node, with the key value range being the range of key values ​​that the model segment is responsible for predicting, and the child node pointers being references to the next lower, more refined model segments.

[0099] Through the above abstraction, all heterogeneous indexes are converted into a list of nodes with the same data structure for subsequent capability labeling and cost estimation.

[0100] Subsequently, the server labels each index metamodel with index capability tags based on the index type and data distribution characteristics. The labeling process is divided into two phases: static labeling and dynamic adjustment. In the static labeling phase, basic capabilities are determined according to the following capability determination rules:

[0101] B-tree index: Possesses point query and range scan capabilities, but lacks sequential write optimization and bitmap optimization capabilities. LSM-Tree index: Possesses point query capabilities, range scan capabilities are weak but still marked as possessing them, possesses sequential write optimization capabilities, but lacks bitmap optimization capabilities. BRIN index: Lacks point query capabilities, possesses range scan capabilities, and only has optimization capabilities when data is written sequentially; lacks bitmap optimization capabilities. Bitmap index: Possesses point query capabilities, lacks range scan and sequential write optimization capabilities, but possesses bitmap optimization capabilities. Learned index: Possesses point query and range scan capabilities, but lacks sequential write optimization and bitmap optimization capabilities.

[0102] Among them, B-Tree leaf nodes support quick record location using equality conditions (such as id = 123); LearnedIndex is also suitable for point query scenarios when prediction errors are controllable; therefore, such nodes can be marked as having point query capabilities. B-tree indexes have ordered linked lists between their leaf nodes, making them suitable for range queries; BRIN indexes are suitable for scanning and filtering large ranges of continuous data; therefore, such nodes can be marked as having range scan capabilities. LSM indexes' MemTable and lower-level SSTable are suitable for high-frequency sequential writes; new data is mainly written in append mode rather than in-place updates; therefore, such nodes can be marked as having sequential write optimization capabilities. Bitmap indexes' bit vector nodes are suitable for fast filtering of low-radix numeric fields (such as status fields and enumeration fields); queries can complete filtering through bitwise operations; therefore, such nodes can be marked as having bitmap optimization capabilities.

[0103] During the dynamic adjustment phase, the static labels are corrected based on the actual data distribution indicators provided by the monitoring module. The dynamic adjustment rules include:

[0104] For bitmap indexes, when the monitoring module detects that the cardinality of the current data segment exceeds a preset cardinality threshold (e.g., 1000), the bitmap optimization capability label is set to "No," because the storage overhead of the bitmap index has already increased significantly. For BRIN indexes, when the monitoring module detects that the orderliness index of the data distribution is lower than a preset orderliness threshold (e.g., 0.3), the range scan capability label is set to "No," because the filtering effect of the BRIN index decreases significantly at this time. For LSM-Tree indexes, when the monitoring module detects that the point query ratio exceeds a preset ratio threshold (e.g., 80%) and the level depth of the data segment exceeds a preset depth threshold (e.g., 3 levels), the point query capability label is downgraded, indicating that although the capability is present, its performance is poor. Dynamic adjustment is achieved through a pre-configured rule engine, which reads relevant metrics from SegmentMetrics, matches each rule, and updates the index capability label accordingly.

[0105] Furthermore, the server labels each index metamodel with an evolutionary capability tag, which is a list of target index types that the index is allowed to be converted into. Evolutionary capabilities are determined based on a predefined directed graph of evolutionary paths. This directed graph is pre-configured by the system administrator based on the feasibility and benefits of index conversion. In this application, the evolutionary path graph is defined as follows: B-tree indexes can evolve into learned indexes or LSM-Tree indexes. LSM-Tree indexes can evolve into B-tree indexes. BRIN indexes can evolve into B-tree indexes. Bitmap indexes can evolve into B-tree indexes. Learned indexes can evolve into B-tree indexes.

[0106] For a given index type, query all outgoing edges from the evolution path graph to obtain a list of convertible target index types, and store it in the evolution capability field. For example, the evolution capability list of a B-tree index is [learning index, LSM-Tree index]; the evolution capability list of an LSM-Tree index is [B-tree index]; and the evolution capability list of a bitmap index is [B-tree index].

[0107] In some cases, even if a path exists in the graph, evolution may be prohibited due to data characteristics. For example, when the cardinality of a field in a bitmap index is extremely low (e.g., less than 10), the bitmap index is already the optimal structure and should not evolve. Therefore, this application can further set evolution filtering rules, which can be based on expert experience. For example, if the current index type is a bitmap index and the cardinality of the data fragment is less than a preset minimum cardinality threshold (e.g., 10), then the list of evolvable capabilities will be cleared.

[0108] The index metamodel also includes storage overhead estimation information and build time estimation information. Specifically, these can be calculated using pre-defined formulas for storage overhead estimation and build time estimation, and then stored as storage overhead estimation information and build time estimation information. The storage overhead estimation formula can be configured with different calculation methods for different index types. For example, for a B-Tree index:

[0109] ;

[0110] in, This is an estimated storage overhead value for the B-Tree index. Indicates the key-value length; 8 represents the 8-byte pointer overhead. For the number of rows in the data segment, This is the preset fill factor.

[0111] For LSM-Tree indexes:

[0112] ;

[0113] in, This is an estimate of the storage overhead for an LSM-Tree index. This is the preset Bloom filter ratio, which defaults to 0.1.

[0114] For BRIN indexes:

[0115] ;

[0116] in, This is an estimated storage overhead value for the BRIN index, where 4 represents the number of bytes of location information. Indicates the number of data blocks.

[0117] For bitmap indexes:

[0118] ;

[0119] in, This represents an estimate of the storage overhead of the bitmap index. Indicates the cardinality of the field.

[0120] For learning indexes:

[0121] ;

[0122] in, This represents the estimated storage overhead of the learning index. This represents the storage space for model parameters, with 0.1 being the preset outlier storage ratio.

[0123] The formula for estimating construction time can be:

[0124] ;

[0125] in, This represents the estimated construction time. Indicates the data size of a data segment. Indicates disk I / O bandwidth. This represents the preset complexity factor. This represents the preset parallel acceleration factor.

[0126] The aforementioned specific parameters, including key-value length, number of rows in a data segment, number of data blocks, field cardinality, model parameter storage space, data size of a data segment, and disk I / O bandwidth, can be extracted by the server from the database system configuration or the storage information corresponding to the data segment; this application does not impose specific limitations on these parameters. Furthermore, the aforementioned formulas for estimating storage overhead and construction time can be adjusted based on actual usage scenarios and expert experience; this application does not impose specific limitations on these parameters.

[0127] By extracting index nodes, labeling static and dynamic capability tags, labeling evolvable capabilities, estimating storage overhead and construction time, the corresponding index identifier, data segment identifier, and index type are determined. This information is then used to build an index metamodel object, stored in the database, and mapped to the corresponding data segment, thus completing the metamodel construction.

[0128] Subsequently, when determining whether evolution is needed, the server will read the current index metamodel used by the data fragment and the metamodel of the candidate target index type, compare their capability tags, storage overhead and construction time, and select the optimal target index type through the evolution cost model in combination with the access characteristics of the data fragment.

[0129] When a data segment is split or merged, the index metamodel module rebuilds the index metamodel for the newly generated data segment. Specifically, when a data segment is split into two sub-segments, each sub-segment inherits the index structure type of the original segment. The server creates new index metamodel objects for the two sub-segments, whose capability tags and evolvability are the same as the original segment, but the storage overhead and construction time need to be re-estimated based on their respective data volumes. When two adjacent data segments are merged into a new segment, the index structure type of the new segment is redefined. Typically, the index type with better performance from the two original segments is selected, or a new decision is made based on the access characteristics after the merge. The specific determination is based on user-defined rules and is not specifically limited here. The index metamodel module creates a new index metamodel object for the new segment, regenerating all fields based on the characteristics of the merged data.

[0130] Furthermore, when a significant change in the access characteristics of a data segment is detected (e.g., the point query ratio increases from 30% to 80%), this application can also re-execute the dynamic adjustment step to update the dynamic part of the capability label, so as to be able to perceive the actual changes in indexing capabilities in a timely manner.

[0131] Based on the above embodiments, this application can achieve a unified abstraction of heterogeneous index structures and establish an index meta-model to provide a data foundation for subsequent automatic index evolution.

[0132] In this embodiment, the server can independently determine whether each data segment meets the preset evolution triggering conditions. A goodness-of-fit value is calculated based on access metrics, and a comprehensive judgment is made using the index meta-model to determine whether the evolution triggering conditions are met. The index meta-model is used to verify the capability adaptation of candidate target index types obtained based on the goodness-of-fit value. For example, access characteristics (point lookup percentage, range lookup percentage, read-write ratio, write rate, etc.) are compared with the capability labels of the current index to determine whether there is a capability mismatch problem for the current index type. A capability mismatch example is: the access characteristics of the current data segment are: point lookup percentage 15%, range lookup percentage 10%, read-write ratio 0.2 (i.e., write operations account for 80%), and high write rate. The current index is a B-Tree. The capability labels of the B-Tree are: point lookup capability is yes, range scan capability is yes, sequential write optimization capability is no, and bitmap optimization capability is no. The server determines through the index meta-model that: since the read-write ratio of 0.2 is lower than the threshold of 0.3, the current B-Tree index does not have sequential write optimization capability, and therefore it is determined to be a capability mismatch.

[0133] This application can also pre-set a neural network identification model for capability mismatch. This neural network identification module is trained based on training samples composed of several historical access feature data and index meta-model information. This application can also set a rule engine corresponding to the comparison rules to quantify the access features into specific access indicators and compare the access indicators with the numerical values ​​of the corresponding dimensions of the index meta-model, thereby determining whether there is a capability mismatch based on the comparison results. In actual use, the server pre-stores the conditions under which the comparison results are judged as capability mismatch for decision-making.

[0134] If a capability mismatch exists, then in step S105, the candidate target index type will be determined by the index meta-model, and the evolutionary cost will be used to determine whether the evolution triggering condition is met.

[0135] S105, if the preset evolution triggering conditions are met, determine the target index type corresponding to the data segment, and convert the data segment from the current index structure to the target index structure corresponding to the target index type through incremental migration.

[0136] In this embodiment of the application, determining whether the evolution triggering condition is met specifically includes:

[0137] Linear fitting is performed on the access metrics of the data segment, and the goodness-of-fit value is calculated. When the goodness-of-fit value is greater than a preset trend threshold, the change in the access metric is determined to be a trend feature; otherwise, it is determined to be a temporary fluctuation, and the determination is terminated. After determining that it is a trend feature, the index meta-model of the current index of the data segment is read to obtain the capability label of the current index. The access feature is compared with the capability label of the current index. If the capability label of the current index does not match the access feature, a capability mismatch is determined. In the case of a capability mismatch, if the popularity score of the data segment is greater than a preset popularity threshold, the system resource utilization rate is less than a preset safety threshold, and the data segment is not within a preset evolution cooling period, the evolution trigger condition is determined to be met.

[0138] The preset popularity threshold and preset safety threshold are set based on actual usage scenarios and are not specifically limited here.

[0139] The formula for calculating the goodness-of-fit value is as follows:

[0140] ;

[0141] in, This represents the actual access metrics within N statistical windows, such as read / write ratio and percentage of point / range queries. These are the fitted values; This represents the average value of the indicators. R² ranges from [0,1]. If the preset trend threshold is set to 0.85, R² > 0.85 indicates a strong trend in load changes, which can be used as a trigger for evolution. R² ≤ 0.85 indicates temporary fluctuations in load changes; even if a single access indicator exceeds the threshold, evolution will not be triggered, and it will only be recorded as "to be observed." This distinguishes between temporary fluctuations and trend changes, preventing oscillations in the index structure's evolution.

[0142] After determining the trend characteristics, the index meta-model will be further combined to determine whether there is a capability mismatch. If a capability mismatch exists, the evolution triggering conditions will be comprehensively determined based on three conditions: popularity score, system resource utilization rate, and evolution cooldown period.

[0143] Simultaneously, when the evolution triggering conditions are met, the server can also determine the target index type based on the access characteristics of the data fragment through a preset index type rule engine. This index type rule engine includes the following determination rules:

[0144] If the write ratio is high (read-write ratio < 0.3) and the read ratio is low, then choose the LSM-Tree index;

[0145] If range queries account for more than 60% of the total queries and the data is distributed sequentially, then choose the BRIN index.

[0146] If the field cardinality is less than 100, then a bitmap index is selected;

[0147] If the data distribution is stable (distribution fluctuation <20%) and queries are frequent, then choose a learning index;

[0148] If random reads account for more than 70%, then a B-tree index should be selected.

[0149] Furthermore, this application sets an evolution suppression period for each data segment to prevent frequent evolution. The duration of the aforementioned preset evolution cooling period is dynamically calculated based on the construction cost and expected performance gains of this evolution. The specific calculation formula is as follows:

[0150] ;

[0151] in, To preset the duration of the evolutionary cooldown period, Based on the cooldown time, The construction cost represents the resource cost required for this evolution, such as the amount of CPU resources consumed. "Expected performance gain" represents the performance gain anticipated in this evolution, which can be specifically based on the pre-defined server's performance gain calculation function. We obtain a and b, which are the preset read weight and write weight, respectively. X and Y represent the read performance gain and write performance gain after this evolution, respectively. The expected performance gain can also be determined by other methods, which are not specifically limited here. The preset evolution cooldown period has a preset upper limit and a preset lower limit, such as a preset lower limit of 300 seconds and an upper limit of 3600 seconds, to avoid the cooldown being too short or too long.

[0152] In one embodiment of this application, when the server determines that the heat score of a data segment is greater than a preset super hotspot threshold, it forcibly terminates the preset evolution cooling period.

[0153] In other words, the cooldown period can be broken by super hot events: when the Segment heat score is >0.9 (1.28 times the evolution trigger threshold), the cooldown period can be forcibly skipped to trigger evolution, allowing a new evolution to be triggered immediately.

[0154] In another embodiment of this application, determining that the evolution triggering condition is met further includes:

[0155] The evolution cost is calculated based on the construction cost, storage overhead, expected performance gains, and the historical average net loss of the data fragments. The formula for calculating the evolution cost is:

[0156] .

[0157] in, As a price for evolution, For index build time, The index occupies storage space. For the expected performance gains of this evolution, Let be the average net loss of the data segment across several historical evolutions, and , The cost of the nth historical evolution, For the actual benefit of the nth historical evolution, For the number of historical evolutions, , , These are preset weight coefficients, which correspond to the importance weights of construction time, storage overhead, and net revenue, respectively.

[0158] When the evolution cost is less than a preset cost threshold, the evolution operation is triggered. At this point, if the server determines that the evolution benefit outweighs the resource cost, it allows the evolution to proceed. This preset cost threshold is set based on the actual use case and is not specifically limited here.

[0159] In this embodiment of the application, incremental migration is used to transform data fragments from the current index structure to the target index structure corresponding to the target index type, specifically including:

[0160] Create the target index structure. Using index nodes as the smallest unit, migrate index nodes incrementally from the current index structure to the target index structure in batches, according to the order determined by node access frequency and key value range continuity. During the migration process, perform double writes for write operations corresponding to the index nodes being migrated, writing to both the current and target index structures simultaneously. After all index node migrations are complete and consistency checks are passed, switch the current index pointer of the data fragment from the current index structure to the target index structure using atomic operations to make the target index structure effective. Release the storage resources occupied by the current index structure if the reference count of the current index structure is detected to be zero.

[0161] After determining that the preset evolution triggering conditions are met and obtaining the target index type, this application will call the corresponding index building interface to create an empty index according to the target index type in the evolution plan. Taking the index node as the smallest unit, the index nodes will be incrementally migrated from the current index structure to the target index structure in batches according to the order determined by the node access popularity and the continuity of the key value range.

[0162] The migration order is determined according to the following rules: nodes are sorted in descending order of access frequency, with hot nodes being migrated first; nodes with adjacent or consecutive key value ranges are grouped to reduce random access and index fragmentation; each group of nodes constitutes a migration batch. Simultaneously, the size of each migration batch is dynamically controlled based on the amount of data covered by each node, with a default limit of no more than 1000 rows of data or no more than 10 index nodes per batch. During the migration process, the system monitors system resource usage in real time. When CPU or IO utilization exceeds a preset limit, such as 70%, the migration rate is automatically reduced (batch intervals are extended) or migration is paused.

[0163] During the migration process, when performing write operations on the index nodes being migrated, the server performs a double write, simultaneously writing to both the current index structure and the target index structure. Specifically: for nodes that have not yet completed migration, write operations are performed on both the current and target indexes; for nodes that have completed migration, write operations are performed on only the target index; all write operations during the migration are recorded in the migration log for fault recovery.

[0164] After all index nodes have migrated, the server performs a consistency check on the target index to ensure that it contains all data and is consistent with the current index. This consistency check is performed by comparing the metadata of the two indexes (such as the total number of records and key value ranges). Once the consistency check passes, an atomic operation switches the current index pointer of the data fragment from the current index structure to the target index structure. Because pointer reads and writes are atomic operations, the query thread either reads the old index pointer or the new index pointer, avoiding intermediate states, thus achieving lock-free switching.

[0165] After an atomic switch is completed, the server releases the storage resources occupied by the current index structure once it confirms that there are no active references to the current index structure. The database maintains a reference count for each index instance, recording the number of query / write operations currently using that index. Query / write operations increment the reference count after obtaining the index pointer and decrement the reference count after the operation is completed.

[0166] Furthermore, this application can detect unexpected or abnormal performance of the target index after a switch, and the system supports rollback operations. During rollback, the current index pointer is switched back to the old index through atomic operations, and the target index is marked as awaiting reclamation.

[0167] The server also includes a consistency and transaction control module. This module records incremental write operations during the evolution process through migration logs and controls the query access path based on the current index version of the data segment, ensuring transaction isolation and data consistency during the evolution process. The migration log records incremental write operations during the evolution process, with each log entry containing information such as operation type (insert, update, delete), key value, and transaction ID. The log is used to perform final completion of the new index before the switch, ensuring data consistency. Index version control maintains version control information for each segment, selecting the correct index access path based on the current index version of the segment during queries.

[0168] The server also includes an index lifecycle management module, which manages the complete lifecycle of an index using a state machine, from creation, observation, stabilization, evolution, cooling, degradation to recycling, and automatically triggers state transitions based on performance metrics. The state machine defines the following index lifecycle states: Created: The index has been created but not yet used; Observed: The index is in the observation period, and the system is collecting performance metrics; Stable: The index is running stably, and its performance meets expectations; Evolving: The index is evolving; CoolDown: The index is in the evolution suppression period, and triggering new evolutions is prohibited; Degraded: The index performance has degraded and needs optimization or recycling; Recycle: The index has been recycled, and storage space has been released. The index lifecycle management module can automatically adjust the index status, such as: hot spot increase + load trend change → evolution (Stable → Evolving); evolution completed → enter the cooling period (Evolving → CoolDown); cooling period ends → return to stable state (CoolDown → Stable); super hot event (heat > 0.9) → skip the cooling period (CoolDown → Evolving); performance degradation → degradation (Stable → Degraded); idle or redundant → recycling (Degraded → Recycle).

[0169] Through the above-described scheme, this application provides an evolutionary database index structure solution that enables evolution. It divides the data table into multiple data segments and manages the index independently for each segment, achieving fine-grained optimization of the index structure. Furthermore, it dynamically partitions or merges data segments based on the similarity of access features corresponding to the data access model, allowing the data segments corresponding to the index structure to change dynamically and simultaneously enhancing the dynamism of the corresponding index structure. Simultaneously, it utilizes an index meta-model to implement conversion decisions for different index structures and uses incremental migration to achieve smooth online evolution of the index structure. This allows the entire evolution process to proceed without downtime or blocking read / write operations, achieving seamless index structure switching without manual intervention. This solves the technical problems of current database index structures being static, having high maintenance costs, and being unable to adaptively evolve their index structures.

[0170] Figure 2 A schematic diagram of an evolutionary system for an evolving database index structure, as provided in this application embodiment, is shown below. Figure 2 As shown, the evolutionary system 200 capable of evolving database index structures includes:

[0171] The partitioning module 201 is used to divide the data table into multiple data segments. The first determining module 202 determines the access characteristics and system resource status of each data segment. The determining execution module 203, based on the access characteristics and system resource status, determines the access characteristic similarity between adjacent data segments, and dynamically splits or merges the data segments according to the access characteristic similarity. The second determining module 204, based on the access characteristics and a preset index meta-model, determines whether each data segment meets a preset evolution trigger condition. The evolution module 205, if the evolution trigger condition is met, determines the target index type corresponding to the data segment, and incrementally migrates the data segment from the current index structure to the target index structure corresponding to the target index type.

[0172] Figure 3 A schematic diagram of the structure of an evolution device capable of evolving a database index structure, as provided in an embodiment of this application, is shown below. Figure 3 As shown, the device includes:

[0173] At least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0174] The data table is divided into multiple data segments. The access characteristics and system resource status of each data segment are determined. Based on the access characteristics and system resource status, the access characteristic similarity between adjacent data segments is determined, and the data segments are dynamically split or merged based on the access characteristic similarity. Based on the access characteristics and a preset index meta-model, it is determined whether each data segment meets the preset evolution trigger conditions. If the evolution trigger conditions are met, the target index type corresponding to the data segment is determined, and the data segment is converted from the current index structure to the target index structure corresponding to the target index type through incremental migration.

[0175] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system and device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0176] The systems, devices, and methods provided in this application are one-to-one correspondences. Therefore, the systems and devices also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and devices will not be repeated here.

[0177] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0178] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An evolution method of an evolvable database index structure, characterized by, The method includes: Divide the data table into multiple data segments; Determining the access characteristics and system resource status of each data segment; specifically including: performing coarse-grained statistics on all data segments at a first preset period and collecting a first preset indicator; filtering corresponding hot data segments according to the first preset indicator, performing fine-grained sampling on the hot data segments at a second preset period and collecting a second preset indicator; the second preset indicator is a fine-grained indicator different from the first preset indicator; pausing the fine-grained sampling when the system resource utilization rate is greater than a preset high load threshold; the access characteristics include at least the first preset indicator and the second preset indicator, and the system resource status includes at least CPU utilization rate and I / O utilization rate; Based on the access characteristics and the system resource status, the access characteristic similarity between adjacent data segments is determined, and the data segments are dynamically split or merged based on the access characteristic similarity. Based on the access characteristics and the preset index meta-model, it is determined whether each data fragment meets the preset evolution triggering condition; wherein, the index meta-model is constructed by abstracting the capability cost information of various heterogeneous index structures, and the index meta-model includes at least: index capability tags, used to identify the applicability of the index structure in point query, range scan, sequential write optimization or bitmap optimization; evolution capability tags, used to indicate the target index type that the index structure is allowed to transform; and storage overhead estimation information and construction time estimation information; If the preset evolution triggering condition is met, the target index type corresponding to the data segment is determined, and the data segment is converted from the current index structure to the target index structure corresponding to the target index type through incremental migration.

2. The method of claim 1, wherein, The step of dynamically splitting or merging the data segments based on the access feature similarity specifically includes: When the similarity of the access features is continuously less than a preset splitting threshold within N consecutive statistical windows, the data segment is split; where N is a natural number greater than 1. When the access feature similarity is continuously greater than the preset merging threshold within N consecutive statistical windows, the data segment is merged; wherein, the preset splitting threshold is less than the preset merging threshold; the splitting and merging are set to be performed when the system resource utilization rate is less than the preset resource threshold, and the minimum data size of the split data segment is not less than the first preset threshold, and the maximum data size of the merged data segment is not greater than the second preset threshold.

3. The method of claim 1, wherein, The determination that the preset evolution triggering condition is met specifically includes: A linear fit is performed on the access index of the data segment, and the goodness-of-fit value is calculated. When the goodness-of-fit value is greater than the preset trend threshold, the change in the access index is determined to be a trend feature; otherwise, it is determined to be a temporary fluctuation and the determination is terminated. After determining the trend feature, when the popularity score of the data segment is greater than the preset popularity threshold, the system resource utilization rate is less than the preset safety threshold, and the data segment is not within the preset evolution cooling period, the evolution triggering condition is determined to be met.

4. The method of claim 3, wherein, The duration of the preset evolution cooldown period is dynamically calculated based on the construction cost and expected performance gains of this evolution. The specific calculation formula is as follows: ； wherein, is a length of a preset evolutionary cooling period, is a base cooling time, is a construction cost, is an expected performance benefit; the length of the preset evolutionary cooling period sets a preset upper limit and a preset lower limit; The method further includes: When the heat score of the data segment exceeds the preset super hotspot threshold, the preset evolution cooling period is forcibly terminated.

5. The method according to claim 3, characterized in that, The determination that the preset evolution triggering condition is met also includes: The evolution cost is calculated based on the construction cost, storage overhead, expected performance gains, and the historical average net loss of the data segment in this evolution. The formula for calculating the evolutionary cost is as follows: ； in, For the evolutionary cost, For index build time, The index occupies storage space. For the expected performance gains of this evolution, The average net loss of the data segment across several historical evolutions is given, and , The cost of the nth historical evolution, For the actual benefit of the nth historical evolution, For the number of historical evolutions, , , These are preset weight coefficients, which correspond to the importance weights of construction time, storage overhead, and net revenue, respectively. When the evolution cost is less than a preset cost threshold, the evolution operation is triggered.

6. The method according to claim 1, characterized in that, The incremental migration process of converting the data fragment from the current index structure to the target index structure corresponding to the target index type specifically includes: Create the target index structure; Using index nodes as the smallest unit, and following the order determined by node access frequency and key value range continuity, the index nodes are incrementally migrated from the current index structure to the target index structure in batches. During the migration process, a double write operation is performed on the write operation corresponding to the index node being migrated, simultaneously writing to both the current index structure and the target index structure; After all the index nodes have been migrated and their consistency has been verified, the current index pointer of the data segment is switched from the current index structure to the target index structure through an atomic operation, so that the target index structure becomes effective. If the reference count of the current index structure is detected to be zero, the storage resources occupied by the current index structure are released.

7. An evolutionary system for an evolving database index structure, characterized in that, The system is capable of executing the evolution method for an evolving database index structure as described in any one of claims 1-6; the system comprises: The partitioning module is used to divide a data table into multiple data segments; The first determining module determines the access characteristics and system resource status of each data segment; The execution module determines the access feature similarity between adjacent data segments based on the access features and the system resource status, and performs dynamic splitting or merging of the data segments based on the access feature similarity. The second determining module is used to determine, based on the access characteristics and the preset index meta-model, whether each of the data segments meets the preset evolution triggering conditions. An evolution module is used to determine the target index type corresponding to the data fragment when the evolution triggering condition is met, and to convert the data fragment from the current index structure to the target index structure corresponding to the target index type through incremental migration.

8. An evolutionary device for an evolving database index structure, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform an evolutionary method for an evolving database index structure as described in any one of claims 1-6.

Images