A learning-based index incremental maintenance method and apparatus for materials databases
By dividing the LSM-tree hierarchy into fixed-size blocks and counting the number of insertions and deletions to determine the PLR segment reuse strategy, the problems of decreased accuracy and high cost of segmented geometric model indexes during dynamic updates in material databases are solved, achieving efficient local maintenance and improving retrieval stability and adaptability.
Patent Information
- Application Number
- CN202611061652.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-16
- Publication Date
- 2026-11-17
AI Technical Summary
When existing technologies dynamically update material databases, the segment boundaries and error ranges of the segmented geometric model index are prone to failure, leading to a decrease in model prediction accuracy. Furthermore, global updates are costly, lack efficient local maintenance mechanisms, and are difficult to adapt to uneven data distribution.
The Partitioned PGM-Index method is used to divide the ordered hierarchy of the LSM-tree into data blocks of fixed size. During the merging process, the number of blocks inserted and deleted is counted to determine the PLR segment reuse strategy: blocks with unchanged content are directly reused, blocks with controlled perturbation are approximately reused by expanding the search window, and reconstruction is triggered only for blocks that exceed the threshold.
It significantly reduces the computational overhead and system resource consumption for maintaining the material database index, improves the stability of material data retrieval, adapts to the dynamic changes in material data, and maintains the stability of index performance.
Smart Images

Figure CN123115129A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database indexing technology, and in particular to a learning-based incremental index maintenance method and apparatus for materials databases. Background Technology
[0002] In materials science research and engineering applications, materials databases have become a crucial infrastructure supporting materials design, performance analysis, and material screening. Materials databases typically store material composition information, preparation process parameters, and test data on mechanical, thermal, and electrical properties. With the continuous accumulation of experimental and computational simulation data, materials databases exhibit continuous data growth and dynamic updates. Simultaneously, the distribution of materials data often exhibits non-uniformity and local variability, which places higher demands on the query efficiency and maintenance performance of the index structure.
[0003] In recent years, learned indexes have been proposed, achieving high-precision prediction of bond locations by modeling the cumulative distribution function (CDF) of bonds as a learning task. The Piecewise Geometric Model Index (PGM Index) is a representative structure, constructing a multi-layer model through piecewise linear regression (PLR). This not only significantly reduces index space usage but also possesses a provable worst-case error bound. However, PGM-Index is primarily designed for static data. As material data is continuously inserted and deleted, its segment boundaries and error ranges may become invalid, leading to a decrease in model prediction accuracy.
[0004] To support dynamic updates of material data, various dynamic learning indexing methods have been proposed, such as Approximate Learned Exponential (ALEX) based on gap arrays, Learned Index with Predicted Positions (LIPP) with node-wise adjusted values, and Disk-optimized Piecewise Geometric Model (DPGM) based on Log-Structured Merge-tree (LSM-tree). While these methods have improved dynamic insertion capabilities to some extent, they still have significant shortcomings in material database scenarios: DPGM and other LSM-tree-based methods require a full reconstruction of the PLR model after each level merge, consuming 85%–90% of the time during the merge process, while the actual changed material data represents only a small proportion, resulting in high update costs. Methods like ALEX and LIPP, oriented towards single-index structures, lack strict worst-case error bounds, which may lead to performance degradation when material data is unevenly distributed. In addition, existing solutions generally lack efficient local maintenance mechanisms and still mainly rely on global model updates or large-scale segment merging, which makes it difficult to effectively reduce the scope of the update's impact.
[0005] To address the aforementioned issues, this invention proposes Partitioned PGM-Index, a dynamic incremental reconstruction method for PGM indexes based on block-based PLR reuse. This method divides each ordered hierarchy into fixed-size data blocks. During the merging process, it counts the number of insertions and deletions in each block to determine the PLR reuse strategy: blocks with unchanged content are directly reused; blocks with controlled perturbations are approximately reused by expanding the search window; and reconstruction is triggered only for blocks exceeding a threshold. Thus, the reconstruction cost is decoupled from the total hierarchy size to the actual number of changing blocks, making it suitable for index maintenance scenarios in material databases where data is continuously growing. Summary of the Invention
[0006] To address the technical problems existing in the DPGM method during LSM-tree merging, such as the global dependency caused by PLR segments predicting the globally ordered position of keys, the insertion and deletion during merging altering the rank of all subsequent keys, rendering the intercept and error window of the old segments ineffective, and thus leading to a full reconstruction of the PLR model, and redundant PLR reconstruction after merging under loads with strong spatial locality, this invention provides a learning-based incremental index maintenance method and apparatus for materials databases. The technical solution is as follows: On the one hand, a learning-based incremental index maintenance method for materials databases is provided, which is implemented by a learning-based incremental index maintenance device. This method includes: Obtain the set of index key values corresponding to the material data records in the material dataset; based on the piecewise linear fitting method, perform block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; Obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learned index structure, merge and schedule according to the set of index key values to be inserted, and count the set of data disturbance in the blocks; Based on the set of data perturbation in the blocks, the block reuse strategy is determined for each block of the initial learning index structure to obtain the block reuse strategy set. Based on the set of block reuse strategies, differentiated local maintenance is performed on each block of the initial learning index structure to obtain the new PLR segment parameters for each block; based on the new PLR segment parameters, the initial learning index structure is updated to obtain the updated learning index structure.
[0007] On the other hand, a learning-based index incremental maintenance apparatus for materials databases is provided. This apparatus is applied to a learning-based index incremental maintenance method for materials databases, and includes: The learning index construction module is used to obtain the set of index key values corresponding to the material data records in the material dataset; based on the piecewise linear fitting method, it performs block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; The merged disturbance statistics module is used to obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learned index structure, it performs merge scheduling according to the set of index key values to be inserted, and counts the set of disturbance amounts of the block data; The reuse strategy determination module is used to determine the reuse strategy for each block of the initial learning index structure based on the set of data perturbation in the blocks, and to obtain a set of block reuse strategies. The index incremental maintenance module is used to perform differentiated local maintenance on each block of the initial learning index structure according to the block reuse strategy set, to obtain the new PLR segment parameters of each block; and to update the initial learning index structure according to the new PLR segment parameters to obtain the updated learning index structure.
[0008] On the other hand, a learning index incremental maintenance device is provided, the learning index incremental maintenance device comprising: a processor; a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for learning index incremental maintenance of a material database.
[0009] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for incremental maintenance of learning indexes for materials databases.
[0010] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention proposes a learning-based incremental index maintenance method for materials databases. By dividing each ordered level of an LSM-tree into fixed-size data blocks, the method counts the number of insertions and deletions in each block during the merging process to define the total disturbance. Based on this, the PLR segment reuse strategy is determined: blocks with unchanged content are directly and accurately reused; blocks with controlled disturbances are approximately reused by expanding the search window while ensuring error bounds; and local reconstruction is triggered only for blocks that exceed the threshold. This significantly reduces the computational overhead and system resource consumption of materials database index maintenance. By designing a localized PLR range in blocks, each block model can be reconstructed independently based only on its internal elements, avoiding frequent large-scale reconstructions caused by global coupling and improving the stability of material data retrieval. The optimal number of PLR segments is guaranteed in the initial construction phase, and independent reconstruction at the block level is achieved in the incremental maintenance phase. This resolves the contradiction between global optimality and local independence, and makes the index structure better adapt to the dynamic changes in the distribution of material data. By combining the above mechanisms, this invention can maintain stable indexing performance under continuous material data updates, and is applicable to scenarios such as material screening, performance analysis, and data-driven material design. It is a low-cost and high-efficiency learning-based incremental index maintenance method oriented towards the characteristics of material database data. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a learning-based index incremental maintenance method for materials databases provided in an embodiment of the present invention; Figure 2 This is a block diagram of a learning-type index incremental maintenance device for a materials database provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a learning-type index incremental maintenance device provided in an embodiment of the present invention. Detailed Implementation
[0013] The technical solution of the present invention will now be described with reference to the accompanying drawings.
[0014] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.
[0015] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.
[0016] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.
[0017] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0018] This invention provides a learning-based incremental index maintenance method for materials databases. This method can be implemented by a learning-based incremental index maintenance device, which can be a terminal or a server. Figure 1 The flowchart shown is for an incremental maintenance method of a learning-based index for a materials database. The processing flow of this method may include the following steps: S1. Obtain the set of index key values corresponding to the material data records in the material dataset; Based on the piecewise linear fitting method, perform block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; Optionally, based on a piecewise linear fitting method, a segmented learning index model is performed according to the material index key value set to obtain an initial learning index structure, including: Sort the set of indexed key values in descending order to obtain an ordered sequence of key values. With the goal of minimizing the number of segments, a global PLR segment set is generated by global OptimalPLR scanning based on preset initial error parameters and ordered key value sequences. Based on the preset block size parameter, the ordered key value sequence is sampled at equal intervals, and a splitter array is constructed. Based on the splitter array, the global PLR segment set is partitioned to obtain the local PLR segment set of each block; Based on the preset secondary error parameters, a secondary piecewise linear regression is performed according to the splitter array to obtain the secondary PGM-Index; An initial learning index structure is constructed based on the splitter array, the set of local PLR segments for each block, and the secondary PGM-Index.
[0019] In one feasible implementation, the present invention is applicable to learning index structures that use predictive models to estimate the sorting position of index key values, and is particularly applicable to learning index structures constructed using piecewise linear models, such as PGM-Index and its dynamically extended structure.
[0020] The index key value is obtained by extracting and encoding the key attribute fields in the material data record. The key attribute fields include, but are not limited to, at least one of the following: material composition identifier, process parameter value, and performance test index.
[0021] The index keys in the material index key set are sorted according to their value to obtain an ordered key value sequence. Based on the preset initial error parameter ε, piecewise linear fitting is performed on the ordered key value sequence to obtain the underlying piecewise linear model. The piecewise linear fitting can be performed using the optimal piecewise linear approximation algorithm. Under the premise of ensuring that the model prediction error in each segment does not exceed the initial error parameter ε, the ordered key value sequence is divided into several continuous data segments with the goal of minimizing the number of segments. Each data segment corresponds to a linear model segment characterized by key value, slope and intercept.
[0022] The initial error parameter ε is pre-set based on the distribution characteristics of material data in the material database and the required query accuracy, and is used to constrain the maximum prediction error range of each layer of piecewise linear models; the block size parameter C is pre-set based on the trade-off between reconstruction efficiency and query efficiency; secondary error parameters... , namely the PGM error bound, is preset according to the trade-off between block search accuracy and secondary index space usage, and is independent of the initial error parameter ε.
[0023] This invention employs a partitioned Piecewise Geometric Model Index (PGM-Index), which introduces a block-level PLR management mechanism based on DPGM. This design retains the LSM-tree merging semantics of DPGM (exponential hierarchical growth with a radix of 8, cascading pairwise merging) and introduces four structural changes: (1) Each ordered level is divided into blocks of equal size (default) (1) each element); (2) the splitter array records the first key of each block, supporting binary search; (3) a secondary PGM-Index is constructed on the splitter array to move the block search from Second drop Next, among them (16 by default); (4) The PLR segments of each block are stored continuously to eliminate indirect addressing by pointers per block.
[0024] First, perform a global OptimalPLR scan on the ordered data, then... Each element samples a splitter key, and the PLR segment is assigned to each block. Segments that cross the boundary are re-anchored to the block's local coordinate system (position 0 corresponds to the block's first element).
[0025] The global PLR post-partitioning strategy ensures the optimal number of PLR segments during the initial construction. At this stage, blocks serve only as containers for segment allocation and coordinate anchoring; the slope of each segment is determined by the global data. During the incremental reconstruction phase, when a block triggers reconstruction, its new PLR segments are independently calculated from the C elements within that block (OptimalPLR). This eliminates the need for global scanning, thus achieving local independence where "each block's model depends only on its C elements." Therefore, global optimality and block independence belong to two separate stages: the former ensures the quality of the initial construction, while the latter ensures the localization of incremental reconstruction.
[0026] S2. Obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learning index structure, merge and schedule according to the set of index key values to be inserted, and count the set of data disturbance in the blocks; Optionally, based on the initial learned index structure, a merging schedule is performed according to the set of index key-value pairs to be inserted, and the perturbation of the partitioned data is statistically analyzed, including: Write the set of index key values to be inserted into the L0 insertion buffer of the initial learned index structure; Based on the preset hierarchical capacity threshold, the merge operation is verified according to the amount of data inserted into the L0 buffer, and the merge operation verification result is obtained. When the merge operation verification result indicates that a merge operation has been triggered, the L0 insertion buffer is merged and scanned with the old level ordered data of the corresponding level in the initial learning index structure to generate a new level ordered data sequence. During the merge process, the number of insertions and deletions in each block are counted simultaneously to obtain the block data perturbation set.
[0027] In one feasible implementation, the corresponding index key values of the material data records to be inserted are written into the L0 insertion buffer of the initial learning index structure; the L0 insertion buffer, as a temporary write area in memory, receives unordered or ordered index key values to be inserted and temporarily stores them at the lowest level.
[0028] Based on a preset hierarchical capacity threshold, the current data volume of the L0 insertion buffer is detected; the hierarchical capacity threshold is preset according to the cardinality growth strategy of the LSM-tree. When the data volume of the L0 insertion buffer has not reached the hierarchical capacity threshold, the merge operation verification result is that the merge operation is not triggered, and the index key value to be inserted continues to be temporarily stored in the L0 insertion buffer.
[0029] When the amount of data in the L0 insertion buffer reaches or exceeds the level capacity threshold, the merge operation verification result triggers the merge operation. At this time, the set of index key values to be inserted in the L0 insertion buffer is merged and scanned with the old level ordered data of the corresponding level in the initial learning index structure to generate a new level ordered data sequence.
[0030] During the merge scan, the old and new ordered data are merged and sorted by key value; the number of insertions and deletions received in each new block are counted simultaneously: the number of insertions is the number of elements in the set of index key values to be inserted that fall within the key value range of the new block, and the number of deletions is the number of elements marked for deletion in the old level block; the block data perturbation is calculated to obtain the block data perturbation set.
[0031] If the amount of data in the target level after merging reaches its level capacity threshold, the cascading merging operation with the previous level will be triggered, and the above merging scan and statistical process will be repeated until the merging operation verification results are all negative and no merging operation is triggered.
[0032] S3. Based on the set of data perturbation in the blocks, determine the block reuse strategy for each block of the initial learning index structure to obtain the block reuse strategy set. Optionally, based on the set of data perturbations in the blocks, a block-by-block reuse strategy is determined for each block of the initial learning index structure to obtain a set of block reuse strategies, including: When the data perturbation of a block is 0, the block corresponding to the data perturbation adopts a precise reuse strategy; When the disturbance of the block data is greater than 0 and not greater than the preset initial error parameter, the block corresponding to the disturbance of the block data adopts an approximate reuse strategy. When the disturbance of the segmented data exceeds the preset initial error parameter, the segment corresponding to the disturbance of the segmented data adopts the triggered reconstruction strategy.
[0033] In one feasible implementation, a block-by-block reuse strategy is determined for each new block based on the set of data perturbation amounts: Precise reuse determination: When the data perturbation Δi of the block is zero, the new block is determined to be precisely reused; Approximate reuse determination: When the data perturbation Δi of the block is greater than zero and not greater than the preset initial error parameter ε, the new block is determined to be approximately reused. Reconstruction Trigger Decision: When the data disturbance Δi of the new block is greater than the preset initial error parameter ε, or when the new block does not have a corresponding old block, the new block is determined to trigger reconstruction. Collect the new PLR segment parameters of each new block and their corresponding reuse strategy identifiers to obtain a block reuse strategy set.
[0034] Theorem 1 (Approximate Multiplexing Error Boundary) Let S be the error bound of S. Data The constructed set of PLR segments. If The key range lies within the same block boundary, and the total number of elements inserted and deleted within the block is... Then for For any key k, the prediction error using S is at most 0. .
[0035] Proof: Let and These represent the local positions of key k in the old and new blocks, respectively. It already exists and is retained in China. The key k, the old PLR segment set S satisfies Because the total number of insertions and deletions within a block is... The local rank change of any preserved key k is at most 1. ,Right now By the triangle inequality, .
[0036] For the newly inserted key k, since it is still located in Within the key range boundaries of the block, the linear segment constructed by OptimalPLR satisfies the following condition for any key within its coverage area: Error bound, therefore Relative to k The prediction error for the insertion position (i.e., the virtual position between the two reserved keys) still does not exceed [a certain value]. The difference between the virtual position and the actual position rank_new(k) is defined by the insertion amount (not exceeding...). Similarly, we can obtain Therefore, the binary search window is expanded to... Coverage can be guaranteed The actual position of any key k in the middle.
[0037] S4. Based on the set of block reuse strategies, perform differentiated local maintenance on each block of the initial learning index structure to obtain the new PLR segment parameters for each block; based on the new PLR segment parameters, update the initial learning index structure to obtain the updated learning index structure.
[0038] Optionally, based on the set of block reuse strategies, differentiated local maintenance is performed on each block of the initial learning index structure to obtain new PLR segment parameters for each block, including: For blocks that are determined to be precisely reused, the PLR segment parameters of the corresponding block are used as the first PLR segment parameters of that block; For blocks that are determined to be approximately reused, the PLR segment parameters of the corresponding block are used as the second PLR segment parameters of that block; For blocks that are determined to trigger reconstruction, piecewise linear regression is performed based on the data set within the corresponding block and the preset initial error parameters to obtain the third PLR segment parameters of the block. The new PLR segment parameters are obtained based on the first PLR segment parameters, the second PLR segment parameters, and the third PLR segment parameters.
[0039] In one feasible implementation, when the content of the new block is completely consistent with the corresponding old block and the cumulative distribution function has not changed, the PLR segment parameters of the corresponding old block are directly used as the new version of the PLR segment parameters of the new block.
[0040] When a new block has local changes, but the position offset of any key does not exceed Δi, the PLR segment parameters of the corresponding old block are used as the new version of the PLR segment parameters of the new block.
[0041] When the set of data elements in the new block is subjected to piecewise linear regression based on the preset initial error parameter ε, the new version of the PLR segment parameters for the new block are obtained.
[0042] The search window corresponding to the parameters of the first PLR segment is 2ε+1; where ε is the preset initial error parameter. The search window corresponding to the parameters of the second PLR segment is 2(ε+Δ). i )+1; where, Δ i This refers to the perturbation amount of the block data; The search window corresponding to the parameters of the third PLR segment is 2ε+1.
[0043] In one feasible implementation, the exact reuse strategy achieves the lowest query cost with a search window of 2ε+1 while avoiding any reconstruction overhead; the approximate reuse strategy slightly widens the search window (from 2ε+1 to 2(ε+Δ)). iThe first option, ε+1), achieves a complete saving of reconstruction overhead; the second option, triggering a reconstruction strategy, restores query efficiency with a standard search window of 2ε+1, but at the cost of partial reconstruction. These three search window designs form a continuous trade-off spectrum between query cost and reconstruction cost, enabling the system to adaptively select the optimal operation path based on the degree of data change.
[0044] In one feasible implementation, based on the above-mentioned block-level PLR reuse mechanism, the following further analyzes the cost optimization and parameter design trade-offs of index maintenance.
[0045] The amortized reconstruction cost analysis of this invention has a hierarchy containing n elements, divided into: Each block. (Note: The original text appears to be incomplete and contains several errors. A more accurate translation would require the full context. To meet the segmentation ratio for reuse conditions, incremental reconstruction only needs to... The cost of recalculating the PLR in blocks proportionally is... And a full reconstruction The above formula only considers the PLR segment reconstruction overhead; fixed costs such as merge scan, splitter sampling, and metadata construction are incurred in both full and incremental merges. The components of the total merge time are reported separately in the experiments. r is determined by the spatial locality of the workload, the insertion scale, and the block size. In the experiments of this invention, r reached 80.5%–84.6% in Hotspot 90 / 10 mode and exceeded 99.9% in Appendix mode.
[0046] The query path cost analysis and block size of this invention With Splitter PGM error boundary The query cost is jointly determined by: (1) for A block lookup of 1 splitter, at a cost of (2) Intra-block segment search (binary search), cost is Increase This will broaden the key range of a single block, increasing the probability of it being touched by an insertion (which is detrimental to precise reuse), but Reducing the size makes the approximate reuse criteria more lenient. The selection of parameters creates a trade-off between query and reconstruction efficiency.
[0047] The delayed build optimization of this invention addresses the issue that for large-scale hierarchies (e.g., approximately 10 million elements, approximately 39,000 blocks), immediately rebuilding the splitter PGM-Index and querying metadata after merging significantly increases insertion path latency. This invention postpones such builds until the first query access to the hierarchical level, shifting the overhead from the insertion path to the query path, thus amortizing the overhead between writes and queries.
[0048] This invention employs a hot / cold metadata separation mechanism for its query path. Each block's query requires access to the slope, intercept, start position, data size, number of segments, minimum / maximum key, and segment offset. This invention splits the metadata into hot and cold parts: HotMeta (16 bytes, cache line aligned) contains the slope, re-anchored intercept, start position, data size, number of segments, and flags; these four records exactly fill a 64-byte cache line. The cold array contains only min_key and max_key, accessed only during negative queries (key-range skip checks). Taking this experimental configuration as an example, the maximum level has approximately 10 million elements. It generates approximately 39,000 blocks at a time, with HotMeta being about 610 KB, which can be fully accommodated in a 1 MB L2 cache.
[0049] The query path in this invention uses Splitter-Key anchoring, and the original segment-by-segment prediction formula is as follows: Anchor key Each block requires 4 bytes of storage. This invention observes that after the splitter binary search is completed, the splitter key... It's already in the CPU cache. Therefore, the intercept is re-anchored to... ,by Replacement without storage HotMeta was compressed from 20 bytes to 16 bytes.
[0050] The query path of this invention uses PGM-accelerated block lookup, which... A naive binary search is required for each splittte. The comparison revealed that the large-step memory access pattern resulted in poor cache behavior. This invention constructs a two-level PGM-Index on the splitter array (…). This narrows the search range to approximately 2. With a window containing 1 element, binary search is reduced to 1. This comparison is performed. PGM prediction and HotMeta speculative prefetching are executed in parallel. Under the configuration of this invention, , At that time, the binary search was reduced from about 15 comparisons to about 5.
[0051] This invention proposes a learning-based incremental index maintenance method for materials databases. By dividing each ordered level of an LSM-tree into fixed-size data blocks, the method counts the number of insertions and deletions in each block during the merging process to define the total disturbance. Based on this, the PLR segment reuse strategy is determined: blocks with unchanged content are directly and accurately reused; blocks with controlled disturbances are approximately reused by expanding the search window while ensuring error bounds; and local reconstruction is triggered only for blocks that exceed the threshold. This significantly reduces the computational overhead and system resource consumption of materials database index maintenance. By designing a localized PLR range in blocks, each block model can be reconstructed independently based only on its internal elements, avoiding frequent large-scale reconstructions caused by global coupling and improving the stability of material data retrieval. The optimal number of PLR segments is guaranteed in the initial construction phase, and independent reconstruction at the block level is achieved in the incremental maintenance phase. This resolves the contradiction between global optimality and local independence, and makes the index structure better adapt to the dynamic changes in the distribution of material data. By combining the above mechanisms, this invention can maintain stable indexing performance under continuous material data updates, and is applicable to scenarios such as material screening, performance analysis, and data-driven material design. It is a low-cost and high-efficiency learning-based incremental index maintenance method oriented towards the characteristics of material database data.
[0052] Figure 2 This is a block diagram of a learning-based index incremental maintenance device for a materials database, provided by an embodiment of the present invention. This device is used for a learning-based index incremental maintenance method for a materials database. (Refer to...) Figure 2 The device includes a learning index construction module 210, a position deviation analysis module 220, an affected area identification module 230, and an asynchronous maintenance execution module 240. Among them: The learning index construction module 210 is used to obtain the set of index key values corresponding to the material data records in the material dataset; based on the piecewise linear fitting method, it performs block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; The merging disturbance statistics module 220 is used to obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learning index structure, it performs merging scheduling according to the set of index key values to be inserted, and counts the set of disturbance amounts of the block data; The reuse strategy determination module 230 is used to determine the reuse strategy for each block of the initial learning index structure based on the set of block data perturbation, and obtain the block reuse strategy set. The index incremental maintenance module 240 is used to perform differentiated local maintenance on each block of the initial learning index structure according to the block reuse strategy set, and obtain the new PLR segment parameters of each block; and update the initial learning index structure according to the new PLR segment parameters to obtain the updated learning index structure.
[0053] Optionally, the learning index building module 210 is further used for: Sort the set of indexed key values in descending order to obtain an ordered sequence of key values. With the goal of minimizing the number of segments, a global PLR segment set is generated by global OptimalPLR scanning based on preset initial error parameters and ordered key value sequences. Based on the preset block size parameter, the ordered key value sequence is sampled at equal intervals, and a splitter array is constructed. Based on the splitter array, the global PLR segment set is partitioned to obtain the local PLR segment set of each block; Based on the preset secondary error parameters, a secondary piecewise linear regression is performed according to the splitter array to obtain the secondary PGM-Index; An initial learning index structure is constructed based on the splitter array, the set of local PLR segments for each block, and the secondary PGM-Index.
[0054] Optionally, the disturbance statistics module 220 is further used for: Write the set of index key values to be inserted into the L0 insertion buffer of the initial learned index structure; Based on the preset hierarchical capacity threshold, the merge operation is verified according to the amount of data inserted into the L0 buffer, and the merge operation verification result is obtained. When the merge operation verification result indicates that a merge operation has been triggered, the L0 insertion buffer is merged and scanned with the old level ordered data of the corresponding level in the initial learning index structure to generate a new level ordered data sequence. During the merge process, the number of insertions and deletions in each block are counted simultaneously to obtain the block data perturbation set.
[0055] Optionally, the reuse strategy determination module 230 is further used for: When the data perturbation of a block is 0, the block corresponding to the data perturbation adopts a precise reuse strategy; When the disturbance of the block data is greater than 0 and not greater than the preset initial error parameter, the block corresponding to the disturbance of the block data adopts an approximate reuse strategy. When the disturbance of the segmented data exceeds the preset initial error parameter, the segment corresponding to the disturbance of the segmented data adopts the triggered reconstruction strategy.
[0056] Optionally, the index incremental maintenance module 240 is further used for: For blocks that are determined to be precisely reused, the PLR segment parameters of the corresponding block are used as the first PLR segment parameters of that block; For blocks that are determined to be approximately reused, the PLR segment parameters of the corresponding block are used as the second PLR segment parameters of that block; For blocks that are determined to trigger reconstruction, piecewise linear regression is performed based on the data set within the corresponding block and the preset initial error parameters to obtain the third PLR segment parameters of the block. The new PLR segment parameters are obtained based on the first PLR segment parameters, the second PLR segment parameters, and the third PLR segment parameters.
[0057] The search window corresponding to the parameters of the first PLR segment is 2ε+1; where ε is the preset initial error parameter. The search window corresponding to the parameters of the second PLR segment is 2(ε+Δ). i )+1; where, Δ i This refers to the perturbation amount of the block data; The search window corresponding to the parameters of the third PLR segment is 2ε+1.
[0058] This invention proposes a learning-based incremental index maintenance method for materials databases. By dividing each ordered level of an LSM-tree into fixed-size data blocks, the method counts the number of insertions and deletions in each block during the merging process to define the total disturbance. Based on this, the PLR segment reuse strategy is determined: blocks with unchanged content are directly and accurately reused; blocks with controlled disturbances are approximately reused by expanding the search window while ensuring error bounds; and local reconstruction is triggered only for blocks that exceed the threshold. This significantly reduces the computational overhead and system resource consumption of materials database index maintenance. By designing a localized PLR range in blocks, each block model can be reconstructed independently based only on its internal elements, avoiding frequent large-scale reconstructions caused by global coupling and improving the stability of material data retrieval. The optimal number of PLR segments is guaranteed in the initial construction phase, and independent reconstruction at the block level is achieved in the incremental maintenance phase. This resolves the contradiction between global optimality and local independence, and makes the index structure better adapt to the dynamic changes in the distribution of material data. By combining the above mechanisms, this invention can maintain stable indexing performance under continuous material data updates, and is applicable to scenarios such as material screening, performance analysis, and data-driven material design. It is a low-cost and high-efficiency learning-based incremental index maintenance method oriented towards the characteristics of material database data.
[0059] Figure 3 This is a schematic diagram of the structure of a learning-type index incremental maintenance device provided in an embodiment of the present invention, as shown below. Figure 3As shown, the learning-type index incremental maintenance device may include the above-mentioned Figure 2 The illustrated learning-based index incremental maintenance device for a materials database. Optionally, the learning-based index incremental maintenance device 310 may include a first processor 2001.
[0060] Optionally, the learning index incremental maintenance device 310 may also include a memory 2002 and a transceiver 2003.
[0061] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.
[0062] The following is combined with Figure 3 A detailed description of each component of the learning-type index incremental maintenance device 310 is provided below: The first processor 2001 is the control center of the learning-type index incremental maintenance device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).
[0063] Optionally, the first processor 2001 can perform various functions of the learning index incremental maintenance device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.
[0064] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.
[0065] In a specific implementation, as one example, the learning-type index incremental maintenance device 310 may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).
[0066] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.
[0067] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the learning index incremental maintenance device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.
[0068] The transceiver 2003 is used to communicate with network devices or with terminal devices.
[0069] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.
[0070] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the learning index incremental maintenance device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.
[0071] It should be noted that, Figure 3 The structure of the learning index incremental maintenance device 310 shown in the figure does not constitute a limitation on the router. The actual learning index incremental maintenance device may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0072] Furthermore, the technical effects of the learning-type index incremental maintenance device 310 can be referred to the technical effects of the learning-type index incremental maintenance method for material databases described in the above method embodiments, and will not be repeated here.
[0073] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.
[0074] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DRRAM).
[0075] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0076] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.
[0077] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.
[0078] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0081] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0083] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A learning-based incremental index maintenance method for materials databases, characterized in that, The method includes: Obtain the set of index key values corresponding to the material data records in the material dataset; based on the piecewise linear fitting method, perform block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; Obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learned index structure, merge and schedule according to the set of index key values to be inserted, and count the set of data disturbance in the blocks; Based on the set of data perturbation in the blocks, the block reuse strategy is determined for each block of the initial learning index structure to obtain the block reuse strategy set. Based on the set of block reuse strategies, differentiated local maintenance is performed on each block of the initial learning index structure to obtain the new PLR segment parameters for each block; based on the new PLR segment parameters, the initial learning index structure is updated to obtain the updated learning index structure.
2. The incremental maintenance method for learning-based indexes of material databases according to claim 1, characterized in that, The piecewise linear fitting method, based on the material index key value set, performs block-based learning index modeling to obtain an initial learning index structure, including: Sort the set of indexed key values in descending order to obtain an ordered sequence of key values. With the goal of minimizing the number of segments, a global PLR segment set is generated by global OptimalPLR scanning based on preset initial error parameters and ordered key value sequences. Based on the preset block size parameter, the ordered key value sequence is sampled at equal intervals, and a splitter array is constructed. Based on the splitter array, the global PLR segment set is partitioned to obtain the local PLR segment set of each block; Based on the preset secondary error parameters, a secondary piecewise linear regression is performed according to the splitter array to obtain the secondary PGM-Index; An initial learning index structure is constructed based on the splitter array, the set of local PLR segments for each block, and the secondary PGM-Index.
3. The incremental maintenance method for learning-based indexes of material databases according to claim 1, characterized in that, The process, based on the initial learned index structure, involves merging and scheduling according to the set of key-value pairs to be inserted, and calculating the disturbance of the partitioned data, including: Write the set of index key values to be inserted into the L0 insertion buffer of the initial learned index structure; Based on the preset hierarchical capacity threshold, the merge operation is verified according to the amount of data inserted into the L0 buffer, and the merge operation verification result is obtained. When the merge operation verification result indicates that a merge operation has been triggered, the L0 insertion buffer is merged and scanned with the old level ordered data of the corresponding level in the initial learning index structure to generate a new level ordered data sequence. During the merge process, the number of insertions and deletions in each block are counted simultaneously to obtain the block data perturbation set.
4. The incremental maintenance method for learning-based indexes of material databases according to claim 1, characterized in that, The step of determining a block-by-block reuse strategy for each block of the initial learning index structure based on the block data perturbation set, and obtaining a block reuse strategy set, includes: When the data perturbation of a block is 0, the block corresponding to the data perturbation adopts a precise reuse strategy; When the disturbance of the block data is greater than 0 and not greater than the preset initial error parameter, the block corresponding to the disturbance of the block data adopts an approximate reuse strategy. When the disturbance of the segmented data exceeds the preset initial error parameter, the segment corresponding to the disturbance of the segmented data adopts the triggered reconstruction strategy.
5. The incremental maintenance method for learning-based indexes of material databases according to claim 1, characterized in that, The step of performing differentiated local maintenance on each block of the initial learning index structure according to the block reuse strategy set to obtain the new PLR segment parameters for each block includes: For blocks that are determined to be precisely reused, the PLR segment parameters of the corresponding block are used as the first PLR segment parameters of that block; For blocks that are determined to be approximately reused, the PLR segment parameters of the corresponding block are used as the second PLR segment parameters of that block; For blocks that are determined to trigger reconstruction, piecewise linear regression is performed based on the data set within the corresponding block and the preset initial error parameters to obtain the third PLR segment parameters of the block. The new PLR segment parameters are obtained based on the first PLR segment parameters, the second PLR segment parameters, and the third PLR segment parameters.
6. The incremental maintenance method for learning-based indexes of material databases according to claim 5, characterized in that, The search window corresponding to the first PLR segment parameter is 2ε+1; where ε is a preset initial error parameter. The search window corresponding to the second PLR segment parameter is 2(ε+Δ i )+1; where, Δ i This refers to the perturbation amount of the block data; The search window corresponding to the parameters of the third PLR segment is 2ε+1.
7. A learning-based incremental index maintenance device for a materials database, wherein the learning-based incremental index maintenance device for a materials database is used to implement the learning-based incremental index maintenance method for a materials database as described in any one of claims 1-6, characterized in that, The device includes: The learning index construction module is used to obtain the set of index key values corresponding to the material data records in the material dataset; based on the piecewise linear fitting method, it performs block-based learning index modeling according to the set of material index key values to obtain the initial learning index structure; The merged disturbance statistics module is used to obtain the set of index key values to be inserted corresponding to the material data record to be inserted; based on the initial learned index structure, it performs merge scheduling according to the set of index key values to be inserted, and counts the set of disturbance amounts of the block data; The reuse strategy determination module is used to determine the reuse strategy for each block of the initial learning index structure based on the set of data perturbation in the blocks, and to obtain a set of block reuse strategies. The index incremental maintenance module is used to perform differentiated local maintenance on each block of the initial learning index structure according to the block reuse strategy set, to obtain the new PLR segment parameters of each block; and to update the initial learning index structure according to the new PLR segment parameters to obtain the updated learning index structure.
8. The learning-type index incremental maintenance device for material databases according to claim 7, characterized in that, The learning index building module is further used for: Sort the set of indexed key values in descending order to obtain an ordered sequence of key values. With the goal of minimizing the number of segments, a global PLR segment set is generated by global OptimalPLR scanning based on preset initial error parameters and ordered key value sequences. Based on the preset block size parameter, the ordered key value sequence is sampled at equal intervals, and a splitter array is constructed. Based on the splitter array, the global PLR segment set is partitioned to obtain the local PLR segment set of each block; Based on the preset secondary error parameters, a secondary piecewise linear regression is performed according to the splitter array to obtain the secondary PGM-Index; An initial learning index structure is constructed based on the splitter array, the set of local PLR segments for each block, and the secondary PGM-Index.
9. A learning-type index incremental maintenance device, characterized in that, The learning-based index incremental maintenance device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.
