A NUMA architecture-aware highly-scalable learning index construction method, system and operating method

Abstract

The application discloses a NUMA architecture-aware highly scalable learning index construction method and system and an operation method. The construction method adopts a hybrid node tree structure to organize data, the structure comprising internal nodes for accurate search in a linear model and leaf nodes for storing data, the leaf nodes being composed of ordered gap nodes and segmented nodes, the segmented nodes supporting dynamic evolution from an ordered phase to a semi-ordered phase based on data density and using hierarchical benchmark buckets to manage conflict data. The system implements an adaptive node evolution mechanism based on an operation cost model, optimizes index performance through node reconstruction triggered by foreground write operations and background hot and cold node compression, adopts a multi-thread concurrent control strategy, and adopts a NUMA-aware memory management and thread scheduling framework. The operation method provides efficient point query and point insertion operation processes. The application can dynamically optimize data layout and reduce operation cost, and is suitable for high-performance database and memory management scenarios.

Description

Technical Field

[0001] This invention belongs to the field of database storage management, specifically relating to a NUMA architecture-aware, highly scalable learning index construction method, system, and operation method. Background Technology

[0002] Indexing technology is fundamental to the efficiency of modern database systems, enabling highly efficient operations. Traditional indexing structures include B-trees, hash indexes, Bloom filters, and skip lists, but they do not consider specific data distributions or characteristics. In recent years, machine learning has developed rapidly, and its ideas and methods have been gradually introduced into database systems.

[0003] The emergence of learned indexing technology has revolutionized index design in Database Management Systems (DBMS), proposing that indexes can be conceptualized as predictive models to predict the location of search keys in a dataset. However, past learned indexing technologies have barely considered the characteristics of the Non-Uniform Memory Access (NUMA) architecture used by high-performance servers, resulting in performance limitations and an inability to adapt to various workloads and deliver high performance under NUMA architectures.

[0004] The NUMA architecture consists of several NUMA nodes, each with its own independent processor, memory space, and PCIe (PCI Express) bus system, and corresponding sockets on the motherboard. The nodes are connected via a high-speed bus to achieve resource aggregation and sharing. In this architecture, memory located on the same NUMA node as the processor is called local memory, while memory on different nodes is called remote memory. Due to the limitations of underlying hardware such as the data bus, processor access to local memory has lower latency and higher bandwidth, while access to remote memory has higher latency and lower bandwidth.

[0005] From the perspective of execution logic and architecture, the performance problems of these learning indexes on NUMA architecture can be attributed to three causal relationships: (1) High memory access cardinality: The operation algorithm generates a large number of memory accesses; (2) High cache miss rate: Frequent cache misses lead to an increased demand for main memory access; (3) NUMA node mismatch: The thread execution operation does not coexist with the NUMA node where the data is located, which exacerbates the problem of remote memory access, which can easily cause latency and limit bandwidth. Summary of the Invention

[0006] Objective: To address the problems in the prior art, this invention proposes a NUMA-aware, highly scalable learning index construction method, system, and operation method. This index construction method dynamically optimizes data storage layout and reduces query and insertion costs; the system provides efficient multi-threading support, significantly improving index performance and scalability under NUMA architecture; the proposed operation method includes efficient point query and point insertion operations. The technical solution of this invention is applicable to high-performance database and memory management scenarios.

[0007] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows:

[0008] This invention provides a NUMA architecture-aware, highly scalable learning index construction method, comprising:

[0009] Construct a hybrid node tree index structure, the structure including internal nodes and leaf nodes, wherein the internal nodes and leaf nodes occupy physical areas of memory respectively;

[0010] The construction of the internal nodes includes: configuring a linear model and establishing a main array to store pointers to the addresses of the next-level nodes;

[0011] The leaf nodes are divided into ordered gap nodes and segmented nodes; the construction of the ordered gap nodes includes: configuring a linear model, establishing a gap array storage structure, arranging the data in the gap array according to the model prediction position, setting a bitmap array to mark the gap state, and filling the gaps with adjacent keywords;

[0012] The construction of the segmented nodes includes: configuring a linear model, dividing the main array into equal-length logical segments, each segment containing a metadata part and a data part, the metadata part recording segment status information, including data volume, segment stage status, concurrency control status, and a segment slot status bitmap, and the data part storing actual data.

[0013] Furthermore, the construction of the segment nodes also includes:

[0014] Set up ordered and semi-ordered stages for segmentation;

[0015] In the ordered phase, the data is arranged in a gap array manner;

[0016] During the semi-order phase, data is hashed into hierarchical buckets.

[0017] The construction of the hierarchical buckets includes:

[0018] Set up a virtual state so that data is stored directly in the main array when there are no conflicts;

[0019] Set the state of the simple buffer array and allocate the buffer array when the number of conflicts is less than the preset value;

[0020] Set the hierarchical baseline bucket status, and allocate a hierarchical baseline bucket when there are more conflicts than the preset value;

[0021] The construction of the hierarchical reference bucket includes:

[0022] Construct a multi-level ordered compact array;

[0023] Configure a maximum of three-level expansion mechanism;

[0024] Configure the baseline layer to store only keywords;

[0025] Ensure that each layer of the structure meets the cache line alignment requirements.

[0026] Furthermore, the method realizes the segmented stage evolution of segmented nodes, the evolution includes an ordered stage and a semi-ordered stage, the data in the ordered stage is arranged according to a linear model, the ordered stage only enters the semi-ordered stage when the data density within the segment reaches a threshold, and the semi-ordered stage is achieved by hashing the data.

[0027] Furthermore, the semi-order stage implements a hierarchical bucket dynamic allocation mechanism. The mechanism is divided into three forms according to the data scale: virtual state, simple buffer array, and hierarchical base bucket. The hierarchical base bucket consists of one or more ordered and compact arrays. The hierarchical base bucket expands layer by layer based on the amount of conflicting data, up to a maximum of three layers. Each layer contains a fixed number of slots and the structure size meets the cache line alignment requirements. The last layer array stores the actual data. The non-bottom layer arrays are called the base layer and only store keywords, which serve as an index to accelerate the process and indicate the data range of the next layer array.

[0028] Furthermore, the method is optimized for NUMA architecture, including data structure localization design and cache line alignment processing.

[0029] This invention provides a NUMA architecture-aware, highly scalable learning index system. The system adopts a hybrid node tree index structure, which includes internal nodes and leaf nodes, wherein the internal nodes and leaf nodes occupy physical areas of memory respectively.

[0030] The internal nodes contain a linear model and a main array storing node pointers, and precise search is achieved through the linear model; the leaf nodes are divided into ordered gap nodes and segmented nodes. The ordered gap nodes use a gap array structure to store data, and the segmented nodes use a logical segmentation structure to store data.

[0031] The indexing system implements an adaptive node evolution mechanism based on an operation cost model;

[0032] The indexing system adopts a hybrid node operation cost model;

[0033] The indexing system implements a multi-threaded concurrency control strategy;

[0034] The system includes a NUMA-aware memory management and thread scheduling framework.

[0035] Furthermore, the adaptive node evolution mechanism includes foreground adjustment and background adjustment;

[0036] The foreground adjustment is performed by the foreground write operation thread, which reduces the operation cost by triggering node evolution, calculating the new node layout, and performing node reconstruction.

[0037] The background adjustment is achieved by having background threads allocated on each NUMA node periodically scan the hot and cold node compression pools on the corresponding NUMA node and compress the nodes to optimize the performance of range operations and the space overhead of indexes.

[0038] The calculation of the new node layout in the front-end adjustment is based on a decentralized node operation counter, combined with a hybrid node operation cost model and evaluated through a fan-out layout decision tree.

[0039] The decentralized node operation counter is used to record the operation increment and operation history information of each thread in the leaf node, including the cumulative number of global operations recorded last time, the number of data nodes in the index, the cumulative number of operation history for each type, and the operation increment count for each thread, etc.

[0040] The centralized hot and cold node compression pool includes a hot scan sparse node pool, a cold access sparse node pool, and a node mapping map shared by the hot and cold node pools. The hot scan sparse node pool stores sparse nodes to be optimized whose range scan operation count exceeds a specified threshold, and the cold access sparse node pool stores sparse nodes to be optimized whose access frequency is lower than a specified threshold.

[0041] Furthermore, the hybrid node operation cost model includes:

[0042] In ordered gap nodes, the cost of query search is estimated based on the error between the actual position of the key and the predicted position of the model, while the cost of insertion operation includes the cost of moving the inserted data in addition to the cost of query search, which is approximated by one-quarter of the square of the length of each continuous dense part of the array in the node.

[0043] The query search cost and insertion cost of a segment node are obtained by adding the costs of each segment;

[0044] The cost calculation for segmentation in the ordered stage is the same as that for the gap array. The cost in the semi-ordered stage is estimated based on the average hierarchical bucket data volume. The average query search cost of the hierarchical bucket is estimated based on the range of the hierarchical bucket binary search. The insertion cost is approximated by one-quarter of the average length of the continuous dense array in the hierarchical bucket.

[0045] The overall operation cost of a node is obtained by multiplying the query cost by the memory access coefficient and the insertion operation cost multiplied by the consecutive memory access coefficient, weighted proportionally.

[0046] When the proportion of write operations in the node's historical workload exceeds a specified threshold, the cost of writing to the ordered gap node will be multiplied by a penalty factor to reduce potential write concurrency conflicts and memory access blocking caused by write amplification in the ordered gap node.

[0047] Furthermore, the multi-threaded concurrency control strategy includes:

[0048] Leaf nodes use basic concurrency locks and read-write lock-based node evolution locks to manage concurrency between non-structured reconstruction operations and between non-structured reconstruction operations and structured reconstruction operations, respectively.

[0049] The basic concurrent lock allows concurrent read operations through optimistic version control, while the mutex lock design allows only one thread to acquire the lock, meaning concurrent read operations, mutual exclusion for write operations, and mutual exclusion for both read and write operations.

[0050] An ordered gap node contains a basic concurrency lock, while a segmented node maintains a basic concurrency lock in the metadata of each segment. Therefore, an ordered gap node allows only one thread to perform a write operation at a time, while a segmented node allows multiple threads to perform write operations on multiple segments within the node at the same time.

[0051] The node evolution lock based on read-write locks manages write operations for non-structural reconstruction in the form of read locks and manages structural reconstruction operations in the form of write locks.

[0052] Furthermore, NUMA-aware memory management calculates the cumulative data size of data nodes on NUMA nodes using a weighted round-robin method based on node resource distribution, and allocates the next data node to the corresponding NUMA node according to the relative baseline coefficient, thus distributing data nodes to each NUMA node;

[0053] The NUMA-aware query reallocation scheduling mechanism dynamically determines the reallocation of query requests based on the difference in computing resources between the NUMA node where the actual leaf node is located and the node where the current thread executing the request is located.

[0054] This invention provides a NUMA architecture-aware, highly scalable learning index operation method, the operation method including a point query operation method and a point insertion operation method;

[0055] The point query operation method includes the following steps:

[0056] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its linear model, until the data node is reached;

[0057] Step 2: If the data node type is ordered gap node, use the linear model within the node for prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends;

[0058] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5.

[0059] Step 4: Based on the initial position predicted by the linear model, take the modulo of the segment size to obtain the segment offset. If the data of this offset does not match the query, use exponential search to correct the prediction error within the segment, obtain the query result, and the query ends.

[0060] Step 5: Obtain the corresponding offset within the segment based on the query keyword hash, and determine the hierarchical bucket status at the current offset according to the slot status bitmap in the segment metadata. If the hierarchical bucket is in a virtual state, directly compare the key stored at the segment offset and return the result based on the matching result. Otherwise, further determine the hierarchical bucket status based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, directly scan the buffer array for matching and return the result based on the matching result. Otherwise, if the hierarchical bucket structure is a hierarchical base bucket, proceed to step 6.

[0061] Step 6: Read the metadata of the hierarchical base bucket to obtain the number of the bucket and the starting address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group in the bottom layer array, obtain the query result, and end the query.

[0062] The point insertion operation method includes the following steps:

[0063] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its linear model, until the data node is reached;

[0064] Step 2: If the data node type is ordered gap node, use the intra-node linear model for prediction, and use exponential search to query the insertion position in the node array as needed, perform move insertion or deletion, update the local insertion and deletion statistics, check the node status, if the current node linear model fails or the data volume triggers the threshold, then go to step 6, otherwise the insertion ends.

[0065] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5.

[0066] Step 4: Based on the initial position predicted by the linear model, take the modulo of the segment size to obtain the segment offset. If the data at this offset cannot meet the data ordering requirements, use exponential search to correct the prediction error within the segment to obtain the insertion position. Find the nearest gap through the bitmap of the metadata and perform the move insertion. If the data volume in the current stage reaches the preset density limit, the segment structure changes from the ordered stage to the semi-ordered stage, the data is redistributed, and the insertion ends.

[0067] Step 5: Obtain the corresponding offset within the segment based on the hash of the inserted key, and determine the hierarchical bucket status at the current offset according to the data slot status bitmap in the metadata. If the hierarchical bucket is in a virtual state, insert directly. If a conflict occurs, request a simple buffer array as the hierarchical bucket, and the insertion ends. If the hierarchical bucket is a simple buffer array, perform simple shift insertion. If the density of the buffer array reaches the preset value, it is converted into a hierarchical base bucket, and the insertion ends. If the hierarchical bucket is a hierarchical base bucket, read the metadata of the hierarchical base bucket, perform a binary search in the base layer to locate the specific group of the bottom array, and perform shift insertion. If there is no gap in the group, find the nearest gap from the neighboring group, shift insertion, and update the base value at the corresponding position in the base layer. If the hierarchical base bucket reaches the preset density limit after insertion, go to step 6; otherwise, the insertion ends.

[0068] Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information, data distribution, and historical operation statistics. The insertion ends.

[0069] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the point query operation method or point insertion operation method of the NUMA architecture-aware highly scalable learning index system as described above.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the point query operation method or point insertion operation method of the NUMA architecture-aware highly scalable learning index system as described above.

[0071] Beneficial effects:

[0072] (1) The present invention adopts a hybrid node tree structure, in which the segmented nodes divide the data through segmentation logic, constrain the operation range, and reduce unnecessary memory access amplification caused by adjacent data conflicts. At the same time, the hierarchical base bucket design maximizes the use of cache line alignment through a compact multi-level array structure, allocates and expands layer by layer as needed, significantly reduces the memory access cost in high memory access scenarios, and solves the problem of high memory access cardinality.

[0073] (2) This invention optimizes data layout and reduces the impact of cache misses on performance through the dynamic evolution mechanism of ordered gap nodes and segmented nodes. Ordered gap nodes combine a linear model and a bitmap array to accurately predict and locate data, reducing the probability of cache misses during access. Segmented nodes dynamically switch between ordered and semi-ordered stages to adapt to different data distribution scenarios, further optimizing local access patterns. Furthermore, the hierarchical baseline bucket design uses hierarchical search to access the underlying actual data only when necessary, effectively reducing the impact of cache misses on performance and solving the problem of high cache miss rates.

[0074] (3) This invention employs NUMA-aware memory management and a weighted round-robin method based on node resource distribution to dynamically allocate data nodes to corresponding NUMA nodes, balancing resource load and reducing the occurrence of remote memory access. Simultaneously, the NUMA-aware query reallocation scheduling mechanism dynamically adjusts the allocation of query tasks according to the resource ratio between the thread and the NUMA node where the data node resides, ensuring that operations are completed locally as much as possible, thus reducing latency caused by remote memory access. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of an indexing system according to an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the leaf node structure layout according to an embodiment of the present invention;

[0077] Figure 3 This is a schematic diagram of the segment layout in the segment node of this invention embodiment;

[0078] Figure 4 This is a schematic diagram of the hot and cold node compression pool according to an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of the foreground adjustment of the adaptive node evolution mechanism in an embodiment of the present invention;

[0080] Figure 6 This is a schematic diagram of the background adjustment of the adaptive node evolution mechanism in an embodiment of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of this invention. After reading this invention, any modifications of this invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0082] This invention belongs to the field of database storage management, specifically relating to a NUMA architecture-aware, highly scalable learning index construction method, system, and operation method. The learning index construction method includes the step of establishing a hybrid node tree index structure, which consists of internal nodes and two types of optimized leaf nodes. The internal nodes achieve accurate searching through a linear model, and the leaf nodes include two types: write-optimized and space-optimized.

[0083] The system includes:

[0084] A hybrid node tree index structure, wherein the internal nodes store address pointers to the next level node, and the leaf nodes store the actual data information;

[0085] An adaptive node evolution mechanism that automatically adjusts the data node layout based on historical access information and data distribution;

[0086] A NUMA-aware memory management and thread task scheduling framework that optimizes resource allocation under NUMA architecture.

[0087] The specific operation method includes:

[0088] A point query operation method, wherein the method locates the target leaf node by traversing the internal nodes and adopts different query strategies according to the leaf node type;

[0089] The point insertion operation method performs data insertion in leaf nodes and triggers an adaptive node evolution mechanism. The technical solution significantly improves the scalability and operational efficiency of the learning index under the NUMA architecture through the synergistic cooperation of the above-mentioned construction method, system structure and operation method.

[0090] Figure 1The diagram shows the structure of the index construction according to an embodiment of the present invention. The index consists of internal nodes and leaf nodes. The internal nodes serve as the index part and do not store actual data. Instead, they store several address pointers pointing to the next level of internal nodes or data nodes, achieving accurate search through a linear model. The leaf nodes store all data information, and adjacent leaf nodes are connected. Leaf nodes consist of two types of nodes: ordered gap nodes and segmented nodes. The ordered gap nodes are organized in the same way as the gap array in the leaf nodes of ALEX, and the data is arranged in order according to the key. The segmented nodes consist of multiple logical segments, and the data between each segment is relatively ordered. Each segment consists of a fixed-length array and several hierarchical buckets. The segments are divided into ordered stages and semi-ordered stages according to the data scale. The ordered stage has no hierarchical buckets, and the data is arranged and organized in the segment array in the manner of the gap array. The semi-ordered stage data is hashed in the hierarchical buckets corresponding to the array within the segment.

[0091] Specifically, in terms of the internal composition of a node, each node contains a linear model consisting of a slope and an intercept of double precision type, as well as basic metadata such as node type, node deletion status, node depth, and the repetition coefficient of the node in its parent node. The linear model is used to calculate the physical storage location (Pos) of the query request key Key in the node, and the calculation logic is Pos = Key × Slope + Intercept. In addition to the basic metadata and internal node metadata (internal node concurrency lock, internal node main array size), internal nodes also contain a main array that stores addresses pointing to the next level of nodes.

[0092] Figure 2 The diagram shows a schematic of the leaf node structure layout in an embodiment of the present invention. The main structure of the ordered gap node consists of a linear model and a main array for storing data (the array is a continuous physical address space in physical memory space, and the slot is the smallest physical storage unit defined by data type in this space). The data in the main array is arranged according to the model prediction (when several keys are predicted to the same slot, the keys are arranged in an ordered manner and stored in the nearest adjacent gap in the entire array). The state of the array slot is indicated by an additional bitmap array. The gaps in the middle are filled with adjacent keys, but the state still belongs to the gap. The main structure of the segment node includes a linear model and a main array for storing data. The main array consists of multiple equal-length segments, each segment having a size of 16 slots. Each segment is divided into an ordered stage and a semi-ordered stage according to the data scale. The data in the ordered stage is arranged in a gap array manner, and the data in the semi-ordered stage is hashed in the buckets corresponding to the array within the segment.

[0093] Figure 3The diagram shows the layout of segments in a segmented node according to an embodiment of the present invention. The segment slots are divided into two parts: a metadata part and a data part. The metadata part occupies one slot (64 bits) and contains status information about the segment, such as the amount of data held, the current segment's stage status, concurrency control status, and other relevant indicators. Specifically, the high 32 bits of the 64-bit metadata field are dedicated to concurrency control, with the most significant bit (MSB) serving as a write exclusive flag. After each write operation, a 31-bit counter increments by 1, and the write exclusive flag is set. The higher 16 bits of the 32-bit metadata represent the amount of data in the current segment, while the lower 16 bits form a status bitmap of the data slot, indicating the status of the data part. The least significant bit (LSB) indicates the segmentation stage, with 1 indicating the first stage.

[0094] Specifically, the first stage of a segment is the ordered stage. To avoid adding extra storage overhead and provide efficient read and write operations, the data in the segment is arranged in the corresponding segment data part based on the model prediction results. Similar to the gap array, gaps exist between data in this stage, which are filled by adjacent keys. The bitmap indicating the slot status reuses the 15 bits after the LSB in the metadata. Data is written and queried primarily according to the position predicted by the model. Only when the key at the corresponding position does not match is a secondary query and insertion performed in the same segment space. When the data density within a segment reaches or exceeds a threshold (default setting is 0.8), the segment transitions to the next stage. The semi-ordered stage segmentation resolves further conflicts caused by the model and insertion through hierarchical buckets. In addition to the key and gap states, the slots in the segment array have hierarchical bucket pointer states. Data is hashed into the corresponding hierarchical buckets in the segment array according to the key hash method.

[0095] Specifically, the second stage of segmentation is the semi-order stage. In this stage, segmentation resolves further conflicts caused by modeling and insertion through hierarchical buckets. In addition to the key and gap states, the slots in the segment's array have hierarchical bucket pointer states. Data is hashed into the corresponding hierarchical buckets within the segment's array using key hashing. To balance space overhead and performance, segmentation employs a capacity-incremental mechanism based on the amount of conflicting data. Hierarchical buckets are dynamically allocated as needed: when there are no keys or key conflicts in a segment's data slots (i.e., fewer than two hashed keys), the data is directly stored in the main array without additional structure; in this case, the hierarchical bucket is in a virtual state. When there are key conflicts in a slot but less than the preset buffer size (default 4), the slot dynamically allocates a subarray (called a simple buffer array, with 5 slots, including 1 metadata slot) as a hierarchical bucket, and the data is compactly and orderly arranged in the simple buffer array. When there are more conflicts, the slot dynamically allocates a larger hierarchical base bucket as a hierarchical bucket, and the data is orderly arranged within it. The hierarchical base bucket expands layer by layer as the scale of conflicting data increases, up to a maximum of three layers. The status of a hierarchical bucket is indicated by a bitmap consisting of the 15 bits following the LSB in the metadata. When the bit is 0, the hierarchical bucket is in a virtual state. For example, if the lower 16 bits of the metadata are 0b0000000000000010, the data slot with segment offset index 1 is dynamically allocated a simple buffer array or a hierarchical base bucket. Hierarchical buckets in other data slots within the segment are in a virtual state and have no actual allocated space. The addresses of hierarchical buckets in non-virtual states are stored in the corresponding data slots. When the hierarchical bucket is a simple buffer array, the highest bit of the data slot address is set to 1 to distinguish the hierarchical bucket's status.

[0096] Specifically, the hierarchical baseline bucket expands layer by layer as the data volume grows, with each layer containing a fixed number of slots. To achieve memory alignment, the array size in the hierarchical baseline bucket is always an integer multiple of the cache line size. The array in the first layer is the size of two cache lines, and the first layer contains metadata for storing data count information, the array address of the associated value data corresponding to the key, and metadata about the existing second and third layer arrays. When a layer is full, the key in that layer is fixed as the baseline value to indicate the search range, and a new array is directly allocated for expansion. The new array is logically divided into several groups of equal length, each containing 8 slots. The number of new arrays is the same as the number of data slots in the array of the previous layer. The size of the second and third layer arrays is an integer multiple of the cache line size. The arrays in the hierarchical baseline bucket that are not at the bottom layer are called the baseline layer, which serves as an index for acceleration. The key value in the baseline layer corresponds to the first key value of the corresponding group in the next layer array, and is also the smallest key value in the group. The hierarchical baseline bucket can expand to a maximum of three layers. Exceeding this limit may indicate that there are too many node data records or a decrease in the accuracy of linear model predictions, leading to a reduction in node performance. Therefore, when the key is a fixed 8-byte size, the first-level array contains 16 slots, the first 4 slots store metadata, the second-level slots have 8 times the number of data slots in the first level, and the third-level slots have 8 times the number of data slots in the second level.

[0097] Specifically, the statistical information structure in the adaptive node evolution mechanism of this invention includes a decentralized node operation counter and a centralized hot and cold node compression pool, providing historical access information and candidate evolved node information for node evolution. During indexing, worker threads are responsible for updating the operation information of relevant nodes, where write operations trigger foreground node adjustments and scan operations trigger background node compression. The background thread then filters and compresses nodes that meet the conditions from the hot and cold node compression pool.

[0098] See Figure 4 The diagram illustrates a hot and cold node compression pool according to an embodiment of the present invention. The hot and cold node compression pool consists of a hot-scanning sparse node pool, a cold-access sparse node pool, and a node mapping map shared by the hot and cold node pools. Each NUMA node maintains its own hot and cold node compression pool. Specifically, the hot-scanning sparse node pool stores frequently accessed and structurally sparse nodes on the current NUMA node, while the cold-access sparse node pool stores structurally sparse nodes with extremely low access frequency. The node pools are implemented using doubly linked lists. When adding new linked list nodes to the hot-scanning sparse node pool and the cold-access sparse node pool, they are added from the tail node. The closer a node is to the head node in the cold-access sparse node pool, the lower its access frequency. Therefore, the background thread scans from the head node, and removes each compressed node from the compression pool.

[0099] Specifically, during the index building phase, each cold access sparse node pool is initially empty. When the data node space overhead utilization rate is lower than a preset threshold, it is added to the cold access sparse node pool on the corresponding NUMA node. When the worker thread performs foreground adjustment, the old node is removed from the hot and cold node compression pool, and the node that meets the conditions is written into the cold access sparse node pool. In addition, before the background thread performs node compression, the relevant linked list nodes will also be removed from the cold access sparse node pool.

[0100] Specifically, the hot-scan sparse node pool is updated when worker threads detect frequent scans on leaf nodes. When a foreground worker thread performs a range scan operation on segmented nodes or sparse ordered gap nodes, and detects that the scan operation increment in the current node has reached a threshold, the thread further calculates the sum of scan operation increments since the last record of the node, as well as the sum of all operation increments since the last record of the node. If the scan operation ratio exceeds the threshold (default 0.5), it attempts to add the current node to the hot-scan sparse node pool of the corresponding hot and cold node compression pool.

[0101] Figure 5 The diagram shown is a front-end adjustment schematic of the adaptive node evolution mechanism in an embodiment of the present invention. The front-end adjustment process includes steps such as triggering node evolution, calculating the new node layout, and performing node reconstruction.

[0102] Specifically, the conditions for triggering node evolution are divided into the following categories according to node type: 1) When a segmented node write operation detects that the data in the local segment involved in the write has reached a preset upper limit; 2) When an ordered gap node write operation detects that the model has failed (i.e., the cost of historical operations far exceeds the expected cost) or the amount of data in the node has reached a preset upper limit.

[0103] Specifically, the node layout is divided into two steps: 1) centralizing the data in the nodes to be adjusted; 2) calculating the lowest cost layout under historical operational load weighting using a hybrid node cost model combined with a fan-out decision tree, such as the number of new nodes, data partitioning, and the type of new nodes. The fan-out decision tree enumerates a binary tree from 1 to a preset maximum fan-out based on the data range of the original nodes. When the data size of the enumerated data partition meets the maximum data node size constraint, the weighted operational costs of the sequence gap nodes and segmented nodes under linear model prediction control obtained through least squares are calculated respectively, and the fan-out partition with the minimum cumulative cost is selected.

[0104] Specifically, the hybrid node operation cost model is approximated as follows: In ordered gap nodes, the cost of query search is estimated based on the error between the actual position of the key and the predicted position of the model, while the cost of insertion operation includes the cost of moving inserted data in addition to the cost of query search, which is approximated by one-quarter of the square of the length of each continuous dense part of the array in the node; the query search cost and insertion cost of segmented nodes are obtained by adding the costs of each segment; the cost calculation of segments in the ordered stage is consistent with the calculation method of gap array, and the cost in the semi-ordered stage is estimated based on the average hierarchical bucket data volume, where the average query search cost of hierarchical bucket is estimated based on the range of hierarchical bucket binary search, and the insertion cost is approximated by one-quarter of the length of the average continuous dense array in the hierarchical bucket; the comprehensive operation cost of a node is obtained by multiplying the query cost by the memory access coefficient and the insertion operation cost multiplied by the continuous memory access coefficient, weighted according to the operation ratio; when the proportion of write operations in the node's historical workload exceeds a certain threshold, the cost of writing to ordered gap nodes will be multiplied by a penalty factor, because of the potential write concurrency conflicts and memory access blocking caused by write amplification in ordered gap nodes.

[0105] Specifically, in the node reconstruction step selected by the foreground thread, the foreground thread directly performs node reconstruction under the requirements of the new node layout. If the number of new nodes is greater than 1, leaf nodes of the relevant type are obtained by splitting the data according to the partition. If the number of split nodes exceeds the repetition cardinality of the original node, the expansion of the parent node is triggered.

[0106] Figure 6 The diagram illustrates the background adjustment of the adaptive node evolution mechanism in this embodiment of the invention. The background thread prioritizes scanning the hot-scan sparse node pool from beginning to end. After reconfirming the validity of nodes (e.g., frequent scanning but few insertions and poor spatial coherence), non-range-friendly nodes that frequently perform range scans are converted into range-friendly nodes. Specifically, from the perspective of node type, the background thread converts sparse segmented data nodes into compact ordered gap nodes. When the background thread scans the cold-access sparse node pool, it calculates the access coefficient of the current node (the ratio of the node's cumulative operations since the last record to the global cumulative operations during the same period). If the access coefficient is less than the reciprocal of the number of leaf nodes, the node is considered a cold-access node. The node's space utilization is further examined. Only when both cold access and low space utilization are met will the node be compressed; otherwise, the node is moved to the end of the cold-access sparse node pool.

[0107] The concurrency management strategy for multi-threaded operations in this embodiment of the invention is as follows:

[0108] Since writing to an ordered gap node involves moving data across the entire array, this node employs a concurrent strategy of optimistic reads and pessimistic writes.

[0109] Specifically, before performing a read operation in an ordered gap node, the node's underlying concurrency control lock is first read. If the current lock is in a write-exclusive state, proceed to step 1 of the query; otherwise, record the current lock state version information. After obtaining the query result, before ending, the current node's underlying concurrency control lock state information is checked again. If it does not match the recorded information, proceed to step 1 of the query; otherwise, the query ends.

[0110] Specifically, before performing a write operation in an ordered gap node, the system first attempts to acquire the node's underlying concurrency control lock. If this fails, proceed to step 1 of the insertion process. After writing data, if node evolution and reconstruction are not triggered, the underlying concurrency control lock is released, and the insertion process ends. If reconstruction is triggered, the underlying concurrency control lock is not released.

[0111] Since segmented nodes only involve a single segment when performing queries and inserts, they employ finer-grained concurrency control. Each segment's metadata maintains a basic concurrency control lock. Normal operations between segments (operations that do not trigger node reconstruction) are concurrent, while the concurrency control logic for operations within a segment is similar to that of ordered gap nodes, i.e., optimistic reads and pessimistic writes are mutually exclusive.

[0112] Specifically, within a segment node, once the query operation locates a segment, it accesses the segment's metadata and concurrent lock status. If the current segment is in a write-exclusive state, it proceeds to step 1 of the query; otherwise, it records the current lock status version information. After obtaining the query result, before ending, it checks the basic concurrent control lock status information of the current segment again. If it does not match the recorded information, it proceeds to step 1 of the query; otherwise, the query ends.

[0113] Specifically, in a segment node, when the insertion operation locates a segment, it first attempts to acquire the basic concurrency control lock for the segment metadata. If this fails, it proceeds to step 1 of the insertion process. After writing data, if node evolution and reconstruction are not triggered, the basic concurrency control lock is released, and the insertion ends. If reconstruction is triggered, the basic concurrency control lock is not released.

[0114] In addition, to ensure that backend node adjustments do not affect frontend query operations, leaf nodes have added node evolution locks based on read-write locks. Frontend query operations only view the basic concurrency locks and are not affected by node evolution locks. Ordinary write operations need to acquire the read lock of the node evolution lock when entering a node and release the read lock of the node evolution lock after leaving. When a thread performs a structure reconstruction operation, it needs to acquire the write lock of the node evolution lock. If it was originally a read lock, it will be upgraded to a write lock.

[0115] In order to fully utilize the characteristics of the NUMA architecture, namely that each NUMA node contains one or more processors and local memory directly connected through the memory controller, and that processors accessing local memory have lower latency and higher bandwidth, while accessing remote memory has higher latency and lower bandwidth, the NUMA-aware memory management and task scheduling framework in this embodiment of the invention adopts a weighted round-robin method based on the distribution of processor computing resources available on the NUMA node and a query reallocation scheduling mechanism. The former requests memory from the memory space of each NUMA node and allocates it to each data node; the latter selectively schedules and reallocates query task requests to threads running on the processors of each NUMA node.

[0116] Specifically, the weighted round-robin method based on node resource distribution is applied during the index batch loading and building phase, allocating data layer nodes to various NUMA nodes. Assume the future thread resource distribution ratio on each node is r. i Let i = 1…n, where n is the number of nodes, and the total percentage is 1. Using the node with the smallest resource percentage as the baseline, the coefficient 'a' of each node relative to the baseline is obtained. i Considering data locality, this framework uses data nodes in the data layer as the unit of division and the amount of data on each data node as the allocation indicator. It assumes that the average range scan size in the future workload is N. rq And the maximum amount of data that a leaf node can store is N. m The minimum of the two values ​​is taken as the reference allocation unit value D. r During the batch loading phase, the cumulative data size D of the data nodes on the current NUMA node is calculated. x If D x Greater than or equal to a i *D r If the value is *0.8, then subsequent data nodes will be allocated to the next NUMA node in a round-robin fashion.

[0117] Specifically, the query-based reallocation scheduling mechanism first assumes there are n NUMA nodes, and each NUMA node is allocated t thread resources. i Let i = 1…n. Each thread has a linked array of n requests to be reassigned, bound to fixed NUMA nodes. Each thread is initially assigned a series of requests. During execution, each thread places a portion of the requests into the reassignment array with a certain probability. After the thread has processed the first batch of data, it will reassign requests in a 1 / t sequence. iThe proportion is obtained from the corresponding reallocation array of threads located on other NUMA nodes, and then executed again. During thread execution, non-range operations are executed directly in the current thread. For range query operations involving a range close to the preset Dr, if the actual NUMA node allocated to the leaf node is inconsistent with the current thread after the search layer is completed and the leaf node is located, i.e., execution and data coordination cannot be achieved, then the difference delta between the thread computing resource ratio on the current node and the actual node is calculated. x If the generated random number between (0,1) is greater than delta x If the current operation performs a reallocation and scheduling, it directly returns the leaf node address and its actual node ID; otherwise, it continues executing the current operation. When a thread executes a secondary allocation request, it first attempts to reuse the address provided in the previous execution. If the address has not yet been marked as deleted, it reuses the address; otherwise, it re-executes the complete search steps from the beginning.

[0118] Specifically, the processing point query steps in this embodiment of the invention are as follows:

[0119] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of the model, until the leaf node, i.e. the data node, is reached.

[0120] Step 2: If the data node type is ordered gap node, use the linear model within the node for prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends;

[0121] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5.

[0122] Step 4: Based on the initial position predicted by the model, take the modulo of the segment size to obtain the segment offset (if a metadata slot is encountered, the offset position is shifted one position to the right). If the data of this offset does not match the query, use exponential search to correct the prediction error within the segment, obtain the query result, and the query ends.

[0123] Step 5: Obtain the corresponding offset within the segment based on the hash of the query keyword (shift the offset one bit to the right when encountering a metadata slot), and determine the hierarchical bucket status at the current offset based on the slot status bitmap in the segment metadata. If the hierarchical bucket is in a virtual state, directly compare the key stored at the segment offset and return the result based on the matching result. Otherwise, further determine the hierarchical bucket status based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, directly scan the buffer array for matching and return the result based on the matching result. Otherwise, if the hierarchical bucket structure is a hierarchical base bucket, proceed to step 6.

[0124] Step 6: First, read the metadata of the hierarchical base bucket to obtain the number of the bucket and the starting address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group in the bottom array, obtain the query result, and end the query.

[0125] Specifically, the point insertion steps in this embodiment of the invention are as follows:

[0126] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of the model, until the data node is reached;

[0127] Step 2: If the data node type is ordered gap node, use the intra-node linear model for prediction, and use exponential search to query the insertion position in the node array as needed, perform move insertion or deletion, update the local insertion and deletion statistics, check the node status, if the current node model fails or the data volume triggers the threshold, then go to step 6, otherwise the insertion ends.

[0128] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5.

[0129] Step 4: Based on the initial position predicted by the model, take the modulo of the segment size to obtain the intra-segment offset. If the data at this offset cannot meet the data ordering requirements, use exponential search to correct the prediction error within the segment to obtain the insertion position. Find the nearest gap through the bitmap of the metadata and perform the move insertion. If the data volume in the current stage reaches the preset density limit, the segment structure changes from an ordered stage to a semi-ordered stage, and the data is redistributed. Insertion ends;

[0130] Step 5: Obtain the corresponding offset within the segment based on the hash of the inserted key, and determine the hierarchical bucket status at the current offset according to the data slot status bitmap in the metadata. If the hierarchical bucket is in a virtual state, insert directly. If a conflict occurs, request a simple buffer array as the hierarchical bucket, and the insertion ends. If the hierarchical bucket is a simple buffer array, perform simple shift insertion. If the density of the buffer array reaches the preset value, it is converted into a hierarchical base bucket, and the insertion ends. If the hierarchical bucket is a hierarchical base bucket, read the metadata of the hierarchical base bucket, perform a binary search in the base layer to locate the specific group of the bottom array, and perform shift insertion. If there is no gap in the group, find the nearest gap from the neighboring group, shift insertion, and update the base value at the corresponding position in the base layer. If the hierarchical base bucket reaches the preset density limit after insertion, go to step 6; otherwise, the insertion ends.

[0131] Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information, data distribution, and historical operation statistics. After that, mark the original node as deleted, and the insertion ends.

Claims

1. A method for constructing a highly scalable learning index that is aware of NUMA architecture, characterized in that, include: Construct a hybrid node tree index structure, the structure including internal nodes and leaf nodes, wherein the internal nodes and leaf nodes occupy physical areas of memory respectively; The construction of the internal nodes includes: configuring a linear model and establishing a main array to store pointers to the addresses of the next-level nodes; The leaf nodes are divided into ordered gap nodes and segmented nodes; the construction of the ordered gap nodes includes: configuring a linear model, establishing a gap array storage structure, arranging the data in the gap array according to the model prediction position, setting a bitmap array to mark the gap state, and filling the gaps with adjacent keywords; The construction of the segmented nodes includes: configuring a linear model, dividing the main array into equal-length logical segments, each segment containing a metadata part and a data part, the metadata part recording segment status information, including data volume, segment stage status, concurrency control status and slot status bitmap within the segment, and the data part storing actual data; NUMA-aware memory management uses a weighted round-robin method based on node resource distribution to calculate the cumulative data size of data nodes on NUMA nodes, allocate the next data node to the corresponding NUMA node according to the relative baseline coefficient, and distribute data nodes to each NUMA node. The NUMA-aware query reallocation scheduling mechanism dynamically determines the reallocation of query requests based on the difference in computing resources between the NUMA node where the actual leaf node is located and the node where the current thread executing the request is located.

2. The method according to claim 1, characterized in that, The construction of the segmented nodes also includes: Set up ordered and semi-ordered stages for segmentation; In the ordered phase, the data is arranged in a gap array manner; During the semi-order phase, data is hashed into hierarchical buckets. The construction of the hierarchical buckets includes: Set up a virtual state so that data is stored directly in the main array when there are no conflicts; Set the state of the simple buffer array and allocate the buffer array when the number of conflicts is less than the preset value; Set the hierarchical baseline bucket status, and allocate a hierarchical baseline bucket when there are more conflicts than the preset value; The construction of the hierarchical reference bucket includes: Construct a multi-level ordered compact array; Configure a maximum of three-level expansion mechanism; Configure the baseline layer to store only keywords; Ensure that each layer of the structure meets the cache line alignment requirements.

3. A NUMA-aware, highly scalable learning indexing system, characterized in that, The system adopts a hybrid node tree index structure, which includes internal nodes and leaf nodes, wherein the internal nodes and leaf nodes occupy physical areas of memory respectively. The internal nodes contain a linear model and a main array storing node pointers, and precise search is achieved through the linear model; the leaf nodes are divided into ordered gap nodes and segmented nodes. The ordered gap nodes use a gap array structure to store data, and the segmented nodes use a logical segmentation structure to store data. The indexing system implements an adaptive node evolution mechanism based on an operation cost model; The indexing system adopts a hybrid node operation cost model; The indexing system implements a multi-threaded concurrency control strategy; The system includes a NUMA-aware memory management and thread scheduling framework; The adaptive node evolution mechanism includes front-end adjustment and back-end adjustment; The foreground adjustment is performed by the foreground write operation thread, which reduces the operation cost by triggering node evolution, calculating the new node layout, and performing node reconstruction. The background adjustment is achieved by having background threads allocated on each NUMA node periodically scan the hot and cold node compression pools on the corresponding NUMA node and compress the nodes to optimize the performance of range operations and the space overhead of indexes. The calculation of the new node layout in the front-end adjustment is based on a decentralized node operation counter, combined with a hybrid node operation cost model and evaluated through a fan-out layout decision tree. The decentralized node operation counter is used to record the operation increment and operation history information of each thread in the leaf node, including the cumulative number of global operations recorded last time, the number of data nodes in the index, the cumulative number of operation history for each type, and the operation increment count for each thread. The centralized hot and cold node compression pool includes a hot scan sparse node pool, a cold access sparse node pool, and a node mapping map shared by the hot and cold node pools. The hot scan sparse node pool stores sparse nodes to be optimized whose range scan operation count exceeds a specified threshold, and the cold access sparse node pool stores sparse nodes to be optimized whose access frequency is lower than a specified threshold.

4. The NUMA architecture-aware, highly scalable learning indexing system according to claim 3, characterized in that, The multi-threaded concurrency control strategy includes: Leaf nodes use basic concurrency locks and read-write lock-based node evolution locks to manage concurrency between non-structured reconstruction operations and between non-structured reconstruction operations and structured reconstruction operations, respectively. The basic concurrent lock allows concurrent read operations through optimistic version control, while the mutex lock design allows only one thread to acquire the lock, meaning concurrent read operations, mutual exclusion for write operations, and mutual exclusion for both read and write operations. An ordered gap node contains a basic concurrency lock, while a segmented node maintains a basic concurrency lock in the metadata of each segment. Therefore, an ordered gap node allows only one thread to perform a write operation at a time, while a segmented node allows multiple threads to perform write operations on multiple segments within the node at the same time. The node evolution lock based on read-write locks manages write operations for non-structural reconstruction in the form of read locks and manages structural reconstruction operations in the form of write locks.

5. The NUMA architecture-aware, highly scalable learning indexing system according to claim 3, characterized in that, The NUMA-aware memory management and thread scheduling framework includes: NUMA-aware memory management uses a weighted round-robin method based on node resource distribution to calculate the cumulative data size of data nodes on NUMA nodes, allocate the next data node to the corresponding NUMA node according to the relative baseline coefficient, and distribute data nodes to each NUMA node. The NUMA-aware query reallocation scheduling mechanism dynamically determines the reallocation of query requests based on the difference in computing resources between the NUMA node where the actual leaf node is located and the node where the current thread executing the request is located.

6. A point query operation method for a NUMA architecture-aware, highly scalable learning index system according to any one of claims 3-5, comprising the following steps: Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its linear model, until the data node is reached; Step 2: If the data node type is ordered gap node, use the linear model within the node for prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends; Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5. Step 4: Based on the initial position predicted by the linear model, take the modulo of the segment size to obtain the segment offset. If the data of this offset does not match the query, use exponential search to correct the prediction error within the segment, obtain the query result, and the query ends. Step 5: Obtain the corresponding offset within the segment based on the query keyword hash, and determine the hierarchical bucket status at the current offset according to the slot status bitmap in the segment metadata. If the hierarchical bucket is in a virtual state, directly compare the key stored at the segment offset and return the result based on the matching result. Otherwise, further determine the hierarchical bucket status based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, directly scan the buffer array for matching and return the result based on the matching result. Otherwise, if the hierarchical bucket structure is a hierarchical base bucket, proceed to step 6. Step 6: Read the metadata of the hierarchical base bucket to obtain the number of the bucket and the starting address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group in the bottom layer array, obtain the query result, and end the query.

7. A point insertion operation method for a NUMA architecture-aware, highly scalable learning indexing system according to any one of claims 3-5, comprising the following steps: Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its linear model, until the data node is reached; Step 2: If the data node type is ordered gap node, use the intra-node linear model for prediction, and use exponential search to query the insertion position in the node array as needed, perform move insertion or deletion, update the local insertion and deletion statistics, check the node status, if the current node linear model fails or the data volume triggers the threshold, then go to step 6, otherwise the insertion ends. Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID by an integer multiple of the segment size, and read the segment's metadata. If the segment is in an ordered stage, proceed to step 4; if the segment is in a semi-ordered stage, proceed to step 5. Step 4: Based on the initial position predicted by the linear model, take the modulo of the segment size to obtain the segment offset. If the data at this offset cannot meet the data ordering requirements, use exponential search to correct the prediction error within the segment to obtain the insertion position. Find the nearest gap through the bitmap of the metadata and perform the move insertion. If the data volume in the current stage reaches the preset density limit, the segment structure changes from the ordered stage to the semi-ordered stage, the data is redistributed, and the insertion ends. Step 5: Obtain the corresponding offset within the segment based on the hash of the inserted key, and determine the hierarchical bucket status at the current offset according to the data slot status bitmap in the metadata. If the hierarchical bucket is in a virtual state, insert directly. If a conflict occurs, request a simple buffer array as the hierarchical bucket, and the insertion ends. If the hierarchical bucket is a simple buffer array, perform simple shift insertion. If the density of the buffer array reaches the preset value, it is converted into a hierarchical base bucket, and the insertion ends. If the hierarchical bucket is a hierarchical base bucket, read the metadata of the hierarchical base bucket, perform a binary search in the base layer to locate the specific group of the bottom array, and perform shift insertion. If there is no gap in the group, find the nearest gap from the neighboring group, shift insertion, and update the base value at the corresponding position in the base layer. If the hierarchical base bucket reaches the preset density limit after insertion, go to step 6; otherwise, the insertion ends. Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information, data distribution, and historical operation statistics. The insertion ends.

8. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the point query operation method or the point insertion operation method of the NUMA architecture-aware highly scalable learning index system as described in claim 6 or 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the point query operation method or point insertion operation method of the NUMA architecture-aware, highly scalable learning index system as described in claim 6 or 7.

Images