Method and system for constructing highly extensible learning index perceived by NUMA (Non Uniform Memory Access) architecture and operation method

Abstract

The invention discloses an NUMA (Non Uniform Memory Access) architecture perceived high-scalability learning index construction method and system and an operation method. According to the construction method, a mixed node tree structure is adopted to organize data, the structure comprises internal nodes accurately searched by a linear model and leaf nodes for storing data, the leaf nodes are composed of ordered gap nodes and segment nodes, and the segment nodes support dynamic evolution from an ordered stage to a semi-ordered stage based on data density. Conflict data is managed using hierarchical benchmark buckets. According to the system, a self-adaptive node evolution mechanism based on an operation cost model is realized, index performance is optimized through node reconstruction triggered by foreground write operation and background hot and cold node compression, a multi-thread concurrency control strategy is adopted, and a memory management and thread scheduling framework perceived by NUMA is adopted. The operation method provides an efficient point query and point insertion operation process. The data layout can be dynamically optimized, the operation cost is reduced, and the method is suitable for high-performance database and memory management scenes.

Description

Technical Field

[0001] The present invention belongs to the field of database storage management, and specifically relates to a NUMA architecture-aware highly scalable learning index construction method, system and operation method. Background Art

[0002] Indexing technology is fundamental to the efficiency of modern database systems, enabling efficient operations. Traditional indexing techniques, such as B-trees, hash indexes, Bloom filters, and skip lists, do not consider specific data distribution or characteristics. In recent years, machine learning has rapidly developed, and its concepts and methods have gradually been incorporated into database systems.

[0003] The emergence of learning indexing technology has revolutionized index design in database management systems (DBMSs), proposing that indexes can be conceptualized as predictive models that predict the location of search keys within a dataset. However, previous learning indexing technologies have largely failed to fully consider the characteristics of the non-uniform memory access (NUMA) architecture used by high-performance servers, resulting in performance scalability and the inability to adapt to diverse workloads and deliver high performance under NUMA architectures.

[0004] The NUMA architecture consists of several NUMA nodes, each with its own independent processor, memory space, and PCI Express (PCI Express) bus system, connected by corresponding sockets on the motherboard. Nodes are connected via high-speed buses, enabling resource aggregation and sharing. In this architecture, memory located in the same NUMA node as the processor is called local memory, while memory located in a different node is called remote memory. Due to hardware conditions such as the underlying data bus, processor access to local memory has lower latency and higher bandwidth, while access to remote memory has higher latency and lower bandwidth.

[0005] From the perspective of execution logic and architecture, the performance issues of these learned indexes on NUMA architecture can be attributed to three layers of causality: (1) high memory access cardinality: the operation algorithm generates a large number of memory accesses; (2) high cache miss rate: frequent cache misses lead to increased demand for main memory access; (3) NUMA node mismatch: the thread execution operation does not co-locate with the NUMA node where the data is located, thereby exacerbating the problem of remote memory access, which is both prone to latency and limits bandwidth. Summary of the Invention

[0006] Purpose of the Invention: To address the aforementioned issues in the prior art, the present invention proposes a NUMA-aware, highly scalable learning index construction method, system, and operation method. This index construction method dynamically optimizes data storage layout and reduces query and insertion costs. The system provides efficient multi-threading support, significantly improving index performance and scalability under NUMA architectures. The proposed operation method includes efficient point query and point insertion operations. The technical solution of the present invention is suitable for high-performance database and memory management scenarios.

[0007] In order to achieve the above object of the invention, the technical solution of the present invention is as follows:

[0008] The present invention provides a NUMA architecture-aware highly scalable learning index construction method, comprising:

[0009] Constructing a hybrid node tree index structure, the structure including internal nodes and leaf nodes, wherein the internal nodes and leaf nodes respectively occupy physical areas of a memory;

[0010] The construction of the internal nodes includes: configuring a linear model, establishing a main array for storing pointers to addresses of nodes in the next layer;

[0011] The leaf nodes are divided into ordered gap nodes and segmented nodes; the construction of the ordered gap nodes includes: configuring a linear model, establishing a gap array storage structure, arranging the data in the gap array according to the model prediction position, and setting a bitmap array to mark the gap state, and the gap is filled with adjacent keywords;

[0012] The construction of the segment node includes: configuring a linear model, dividing the main array into logical segments of equal length, each segment contains a metadata part and a data part, the metadata part records the segment status information, including data volume, segment stage status, concurrency control status and intra-segment slot status bitmap, and the data part stores actual data.

[0013] Furthermore, the construction of the segment node further includes:

[0014] Set the ordered and semi-ordered stages of the segmentation;

[0015] In the ordered phase, data are arranged in a gap array manner;

[0016] In the semi-order phase, data is hashed into hierarchical buckets;

[0017] The construction of the hierarchical bucket includes:

[0018] Set the virtual state so that the data is directly stored in the main array when there is no conflict;

[0019] Set the simple buffer array state and allocate the buffer array when the conflict is less than the preset value;

[0020] Set the hierarchical base bucket status and allocate the hierarchical base bucket when the number of conflicts exceeds the preset value;

[0021] The construction of the hierarchical benchmark bucket includes:

[0022] Construct multi-layer ordered compact arrays;

[0023] Set up to three layers of expansion mechanism;

[0024] Configure the baseline layer to store only keywords;

[0025] Ensure that each layer of the structure meets cache line alignment requirements.

[0026] Furthermore, the method realizes segmented phased evolution of segmented nodes, wherein the evolution includes an ordered phase and a semi-ordered phase. The data in the ordered phase is arranged according to a linear model. The ordered phase enters the semi-ordered phase only when the data density in the segment reaches a threshold. The semi-ordered phase is achieved by hashing the data.

[0027] Furthermore, the semi-order stage implements a hierarchical bucket dynamic allocation mechanism, which is divided into three forms according to the data scale: virtual state, simple buffer array and hierarchical reference bucket. The hierarchical reference bucket is composed of one or more layers of ordered and compact arrays. The hierarchical reference bucket is expanded layer by layer based on the amount of conflicting data, and is expanded to a maximum of three layers. Each layer contains a fixed number of slots and the structure size meets the cache line alignment requirements. The last layer of array stores actual data. The non-bottom layer array is called the reference layer and only stores keywords, which plays an index acceleration role and indicates the data range of the next layer of array.

[0028] Furthermore, the method is optimized for the NUMA architecture, and the optimization includes data structure localization design and cache line alignment processing.

[0029] The present invention provides a NUMA architecture-aware highly scalable learning indexing system, wherein the system adopts a hybrid node tree indexing structure, wherein the structure includes internal nodes and leaf nodes, wherein the internal nodes and leaf nodes respectively occupy physical areas of a memory;

[0030] The internal nodes contain a linear model and a main array storing node pointers, and accurate search is achieved through the linear model; the leaf nodes are divided into ordered gap nodes and segment nodes, the ordered gap nodes use a gap array structure to store data, and the segment nodes use a logical segment structure to store data;

[0031] The indexing system implements an adaptive node evolution mechanism based on an operation cost model;

[0032] The indexing system adopts a hybrid node operation cost model;

[0033] The indexing system implements a multi-threaded concurrency control strategy;

[0034] The system includes a NUMA-aware memory management and thread scheduling framework.

[0035] Furthermore, the adaptive node evolution mechanism includes foreground adjustment and background adjustment;

[0036] The foreground adjustment is performed by the foreground write operation thread, which reduces the operation cost by triggering node evolution, calculating the new node layout and performing node reconstruction;

[0037] The background adjustment is performed by a background thread allocated on each NUMA node to periodically scan the hot and cold node compression pools on the corresponding NUMA node and compress the nodes to optimize the performance of range operations and the space overhead of indexes;

[0038] The new node layout calculation in the foreground adjustment is based on a decentralized node operation counter and is obtained through a fan-out layout decision tree evaluation in combination with a hybrid node operation cost model;

[0039] The decentralized node operation counter is used to record the operation increment and operation history information of each thread in the leaf node, including the cumulative number of global operations recorded last time and the number of data nodes in the index, the cumulative number of each type of operation history, and the operation increment count under each thread;

[0040] The centralized hot and cold node compression pool includes a hot scan sparse node pool, a cold access sparse node pool, and a node mapping map shared by the hot and cold node pools, wherein the hot scan sparse node pool stores sparse nodes to be optimized whose range scan operation times are higher than a specified threshold, and the cold access sparse node pool stores sparse nodes to be optimized whose access frequency is lower than a specified threshold.

[0041] Furthermore, the hybrid node operation cost model includes:

[0042] In ordered gap nodes, the cost of query search is estimated based on the error between the actual position of the key and the model's predicted position, while the cost of insert operation also includes the cost of inserting data movement on top of the query search, which is approximated by the sum of one-quarter of the square of the length of each continuous dense part of the array in the node;

[0043] The query search cost and insertion cost of a segment node are obtained by adding the costs of each segment;

[0044] The cost calculation method for the segmentation in the ordered phase is the same as that for the gap array. The cost in the semi-ordered phase is estimated based on the average amount of data in the level bucket. The average query search cost for the level bucket is estimated based on the range of the binary search for the level bucket. The insertion cost is approximated by one-quarter of the average length of the continuous dense array in the level bucket.

[0045] The comprehensive operation cost of a node is obtained by multiplying the query cost by the memory access coefficient and the insertion operation cost by the continuous memory access coefficient, weighted by the operation ratio;

[0046] When the proportion of write operations in the node's historical workload exceeds a specified threshold, the cost of writing to the ordered gap node will be multiplied by a penalty factor to reduce potential write concurrency conflicts on the ordered gap node and memory access congestion caused by write amplification.

[0047] Furthermore, the multi-threaded concurrency control strategy includes:

[0048] Leaf nodes use basic concurrency locks and node evolution locks based on read-write locks to manage concurrency between non-structural reconstruction operations and concurrency between non-structural reconstruction operations and structural reconstruction operations respectively;

[0049] The basic concurrent lock allows concurrent read operations through optimistic version control, and the mutual exclusion lock design only allows one thread to acquire the lock, that is, read operations are concurrent, write operations are mutually exclusive, and read and write operations are mutually exclusive;

[0050] Ordered gap nodes contain a basic concurrency lock, while segmented nodes maintain a basic concurrency lock in the metadata of each segment. Therefore, ordered gap nodes only allow one thread to perform write operations at the same time, while segmented nodes allow multiple threads to perform write operations on multiple segments within the node at the same time.

[0051] The node evolution lock based on the read-write lock manages non-structural reconstruction write operations in the form of a read lock and manages structural reconstruction operations in the form of a write lock.

[0052] Furthermore, NUMA-aware memory management calculates the data size of the accumulated data nodes on the NUMA nodes through a weighted round-robin method based on node resource distribution, allocates the next data node to the corresponding NUMA node according to the relative baseline coefficient, and distributes the data nodes to each NUMA node;

[0053] The NUMA-aware query reallocation scheduling mechanism dynamically determines the reallocation of query requests based on the difference in computing resource ratio between the NUMA node where the actual leaf node is located and the node where the current thread executing the request is located.

[0054] The present invention provides a NUMA architecture-aware highly scalable learning indexing operation method, the operation method including a point query operation method and a point insertion operation method;

[0055] The point query operation method comprises the following steps:

[0056] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of their linear models until reaching the data node;

[0057] Step 2: If the data node type is an ordered gap node, use the linear model within the node to make a prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends;

[0058] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5;

[0059] Step 4: The initial position predicted by the linear model is modulo the segment size to obtain the intra-segment offset. If the offset data does not match the query, the exponential search is used to correct the prediction error within the segment to obtain the query result, and the query ends.

[0060] Step 5: Obtain the corresponding offset within the segment based on the query keyword hash, and determine the hierarchical bucket status at the current offset based on the slot status bitmap in the segment meta-information. If the hierarchical bucket is in a virtual state, directly compare it with the key stored at the segment offset and return the result based on the match. Otherwise, further determine the hierarchical bucket status based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, directly scan the buffer array for a match and return the result based on the match. Otherwise, the hierarchical bucket structure is a hierarchical base bucket, and proceed to step 6.

[0061] Step 6: Read the metadata of the hierarchical base bucket to obtain the bucket layer number and the first address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group of the bottom layer array, obtain the query result, and the query ends.

[0062] The point insertion operation method comprises the following steps:

[0063] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of their linear models until reaching the data node;

[0064] Step 2: If the data node type is an ordered gap node, use the intra-node linear model for prediction and use exponential search to query the insertion position in the node array as needed. Perform move insertion or deletion, update local insertion and deletion statistics, and check the node status. If the current node linear model fails or the data volume triggers the threshold, go to step 6; otherwise, the insertion ends.

[0065] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5;

[0066] Step 4: The initial position predicted by the linear model is modulo the segment size to obtain the intra-segment offset. If the data at this offset cannot meet the data order requirement, an exponential search is used to correct the prediction error within the segment to obtain the insertion position. The nearest gap is found through the metadata bitmap to perform the move insertion. If the current stage data volume reaches the preset density limit, the segment structure changes from the ordered stage to the semi-ordered stage, the data is redistributed, and the insertion ends.

[0067] Step 5: Based on the inserted keyword hash, the corresponding offset in the segment is obtained, and the state of the hierarchical bucket at the current offset is determined according to the data slot state bitmap in the metadata. If the hierarchical bucket is in a virtual state, it is directly inserted. If a conflict occurs, a simple buffer array is requested as the hierarchical bucket, and the insertion is completed. If the hierarchical bucket is a simple buffer array, simple moving insertion is performed. If the buffer array density reaches the preset value, it is converted to a hierarchical reference bucket, and the insertion is completed. If the hierarchical bucket is a hierarchical reference bucket, the metadata of the hierarchical reference bucket is read, a binary search is performed in the reference layer to locate the specific group of the underlying array, and a moving insertion is performed. If there is no gap in the group, the nearest gap is found from the adjacent group, the moving insertion is performed, and the benchmark value of the corresponding position in the benchmark layer is updated. If it is found that the hierarchical reference bucket reaches the preset density upper limit after the insertion, go to step 6, otherwise the insertion is completed.

[0068] Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information combined with data distribution and historical operation statistics. The insertion is completed.

[0069] The present invention also provides an electronic device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the point query operation method or the point insertion operation method of the NUMA architecture-aware highly scalable learning indexing system as described above are implemented.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the point query operation method or the point insertion operation method of the NUMA architecture-aware highly scalable learning indexing system as described above.

[0071] Beneficial effects:

[0072] (1) The present invention adopts a hybrid node tree structure, in which segment nodes divide data through segment logic, constrain the operation range, and reduce unnecessary memory access amplification caused by adjacent data conflicts. At the same time, the hierarchical reference bucket design maximizes the use of cache line alignment through a compact multi-layer array structure, on-demand allocation and layer-by-layer expansion, significantly reducing the memory access cost in high memory access scenarios and solving the high memory access cardinality problem;

[0073] (2) The present invention optimizes data layout through the dynamic evolution mechanism of ordered gap nodes and segmented nodes, reducing the impact of cache misses on performance. Ordered gap nodes are combined with linear models and bitmap arrays, and the probability of cache misses during access is reduced through accurate model prediction and positioning. Segmented nodes dynamically switch between the ordered and semi-ordered stages to adapt to different data distribution scenarios and further optimize local access patterns. In addition, the hierarchical benchmark bucket design uses hierarchical search to access the underlying actual data only when necessary, thereby effectively reducing the impact of cache misses on performance and solving the problem of high cache miss rates.

[0074] (3) This invention uses NUMA-aware memory management and a weighted round-robin approach based on node resource distribution to dynamically assign data nodes to corresponding NUMA nodes, balancing resource loads and reducing the occurrence of remote memory accesses. Furthermore, a NUMA-aware query redistribution scheduling mechanism dynamically adjusts the allocation of query tasks based on the resource ratio between threads and the NUMA nodes where data nodes reside, enabling operations to be completed locally as much as possible, reducing the latency associated with remote memory accesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Schematic diagram of an indexing system according to an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the leaf node structure layout according to an embodiment of the present invention;

[0077] Figure 3 Schematic diagram of the layout of segments in a segment node according to an embodiment of the present invention;

[0078] Figure 4 This is a schematic diagram of the hot and cold node compression pools according to an embodiment of the present invention;

[0079] Figure 5 A schematic diagram of foreground adjustment of the adaptive node evolution mechanism according to an embodiment of the present invention;

[0080] Figure 6 Schematic diagram of background adjustment of the adaptive node evolution mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described clearly and completely below with reference to specific embodiments and corresponding drawings. It should be understood that these embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. After reading this disclosure, modifications of various equivalent forms of the present invention made by those skilled in the art fall within the scope defined by the appended claims.

[0082] The present invention belongs to the field of database storage management, and specifically relates to a NUMA architecture-aware, highly scalable learning index construction method, system, and operation method. The learning index construction method includes the steps of establishing a hybrid node tree index structure, which consists of internal nodes and two types of optimized leaf nodes. The internal nodes are searched accurately using a linear model, and the leaf nodes are of two types: write-optimized and space-optimized.

[0083] The system comprises:

[0084] A hybrid node tree index structure, wherein the internal nodes in the structure store address pointers to the nodes in the next layer, and the leaf nodes store actual data information;

[0085] Adaptive node evolution mechanism, which automatically adjusts data node layout based on historical access information and data distribution;

[0086] A NUMA-aware memory management and thread task scheduling framework is provided, which optimizes resource allocation under the NUMA architecture.

[0087] The operation method specifically includes:

[0088] A point query operation method, wherein the method locates the target leaf node by traversing the internal nodes and adopts different query strategies according to the leaf node type;

[0089] A point insertion operation method, which performs data insertion in leaf nodes and triggers an adaptive node evolution mechanism; the technical solution significantly improves the scalability and operational efficiency of learning indexes under the NUMA architecture through the coordinated cooperation of the above-mentioned construction method, system structure and operation method.

[0090] Figure 1The figure shows a schematic diagram of the structure of the index construction of an embodiment of the present invention. The index consists of internal nodes and leaf nodes. The internal nodes serve as the index part and do not store actual data. Instead, they store several address pointers pointing to the internal nodes or data nodes of the next layer, and achieve precise search through a linear model. The leaf nodes store all data information. Adjacent leaf nodes are connected. The leaf nodes are composed of two types of nodes: ordered gap nodes and segment nodes. The ordered gap nodes are organized in the same way as the gap arrays in the leaf nodes in ALEX, and the data are arranged in order according to the keywords. The segment nodes are composed of multiple logical segments. The data between each segment is relatively ordered. The segment consists of an array of fixed length and several hierarchical buckets. The segment is divided into an ordered stage and a semi-ordered stage according to the data size. The ordered stage has no hierarchical buckets, and the data is arranged and organized in the segment array in the manner of a gap array. The data in the semi-ordered stage is hashed in the hierarchical buckets corresponding to the array in the segment.

[0091] Specifically, in terms of the internal composition of the node, each node contains a linear model consisting of a double-precision slope (Slope) and an intercept (Intercept) and basic meta-information, such as the node type, node deletion status, node depth, and the node's repetition coefficient in its parent node. The linear model is used to calculate the physical storage position (Pos) corresponding to the query request key Key in the node, and the calculation logic is Pos = Key × Slope + Intercept. In addition to basic meta-information and internal node meta-information (internal node concurrency lock, internal node main array size), the internal node also contains a main array that stores the address pointing to the next layer of nodes.

[0092] Figure 2 The figure shows a schematic diagram of the leaf node structure layout of an embodiment of the present invention. The main structure of the ordered gap node is a linear model and a main array for storing data (the array is a continuous physical address space in the physical memory space, and the slot is the smallest physical storage unit defined by the data type in this space). The data in the main array is arranged according to the model prediction (when several keys are predicted to the same slot, the keys are arranged in order and stored in the nearest adjacent gaps in the full array). The status of the array slot is indicated by an additional bitmap array. The middle gap is filled with adjacent keywords, but the status is still that of a gap; the main structure of the segment node includes a linear model and a main array for storing data, wherein the main array consists of multiple equal-length segments, each segment size is 16 slots, and each segment is divided into an ordered stage and a semi-ordered stage according to the data scale. The data in the ordered stage is arranged in a gap array manner, and the data in the semi-ordered stage is hashed in the buckets of each level corresponding to the array in the segment.

[0093] Figure 3The figure shows a schematic diagram of the layout of the segments in the segment node of an embodiment of the present invention. The slot of the segment is divided into two parts: the metadata part and the data part. The metadata part occupies one slot (64 bits) and contains status information about the segment, such as the amount of data held, the stage status of the current segment, the concurrency control status and other related indicators. Specifically, the upper 32 bits of the 64-bit metadata field are dedicated to concurrency control, where the most significant bit (MSB) serves as a write-exclusive flag. After each write operation, the 31-bit counter is incremented by 1 and the write-exclusive bit is set. The upper 16 bits of the metadata 32 bits represent the amount of data of the current segment, while the lower 16 bits are the status bitmap of the data slot, indicating the status of the data part, where the least significant bit (LSB) indicates the segment stage, and 1 indicates the first stage.

[0094] Specifically, the first stage of the segment is an ordered stage. In order to avoid adding additional storage overhead and provide efficient reading and writing, the data in the segment is arranged in the corresponding segment data part based on the results of the model prediction. Like the gap array, there are gaps between the data in this stage, and the gaps are filled with adjacent keywords. The bitmap indicating the slot status reuses the 15 bits after the LSB in the metadata. Data is written and queried first according to the location predicted by the model calculation. Only when the key at the corresponding position does not match, a secondary query and insertion are performed on the space of the same segment. When the data density in the segment reaches or exceeds the threshold (the default setting is 0.8), the segment transitions to the next stage. The segmentation in the semi-order stage resolves more conflicts caused by the model and insertion through hierarchical buckets. In addition to the status of keywords and gaps, the slots of the array in the segment add the status of hierarchical bucket pointers. The data is hashed in the hierarchical buckets corresponding to the array in the segment according to the keyword hashing method.

[0095] Specifically, the second phase of segmentation is the semi-order phase. Segmentation in this phase resolves conflicts caused by modeling and insertions through hierarchical buckets. In addition to key and gap states, slots in the segment array are augmented with hierarchical bucket pointer states. Data is hashed into the corresponding hierarchical buckets in the segment array using a key hashing scheme. To strike a balance between space overhead and performance, segmentation employs a capacity-progressive mechanism based on the amount of conflicting data. Hierarchical buckets are dynamically allocated on demand. When a segment's data slot contains no key or no key conflict (i.e., fewer than two hashed keys), the data is stored directly in the main array without requiring additional structure. In this state, the hierarchical bucket is said to be in a virtual state. When a slot has a key conflict but fewer than a preset buffer size (four by default), a subarray (called a simple buffer array, with five slots, including one metadata slot) is dynamically allocated as a hierarchical bucket. Data is compactly and orderly arranged in the simple buffer array. When conflicts increase, a larger hierarchical base bucket is dynamically allocated as a hierarchical bucket. Data is then ordered and arranged in this bucket. The hierarchical base bucket expands layer by layer as the amount of conflicting data increases, up to a maximum of three layers. The status of a hierarchy bucket is indicated by a bitmap consisting of the 15 bits following the LSB in the metadata. When a bit is 0, the hierarchy bucket is in a virtual state. For example, if the lower 16 bits of the metadata are 0b0000000000000010, the data slot with segment offset subscript 1 is dynamically allocated with a simple buffer array or a hierarchical base bucket. The hierarchy buckets in other data slots within the segment are in a virtual state, with no actual allocated space. The addresses of non-virtual hierarchy buckets are stored in the corresponding data slots. When a hierarchy bucket is a simple buffer array, the most significant bit of the data slot address is set to 1 to distinguish the hierarchy bucket status.

[0096] Specifically, the hierarchical baseline bucket expands layer by layer as data volume grows, with each layer containing a fixed number of slots. To achieve memory alignment, the array size in the hierarchical baseline bucket is an integer multiple of cache lines. The first-level array is two cache lines long. The first level contains metadata for storing data count information, the array address of the key's associated value, and information about existing second- and third-level arrays. When a level is full, the key in that level is fixed to the baseline value to indicate the search range, and a new array is allocated for expansion. The new array is logically divided into groups of equal length, each containing eight slots. The number of slots in the new array is the same as the number of data slots in the previous level array. The second and third level arrays are sized to be integer multiples of cache lines. The non-bottom-level arrays in the hierarchical baseline bucket are called the baseline level and serve as index acceleration. The key value in the baseline level corresponds to the first key value of the corresponding group in the next-level array and is the smallest key value within the group. The hierarchical baseline bucket expands to a maximum of three levels. Exceeding this limit may indicate excessive node data records or decreased linear model prediction accuracy, leading to reduced node performance. Therefore, when the keyword is a fixed 8-byte size, the first-level array contains 16 slots, the first 4 slots store metadata, the number of second-level slots is 8 times the number of first-level data slots, and the number of third-level slots is 8 times the number of second-level data slots.

[0097] Specifically, the statistical information-related structures in the adaptive node evolution mechanism of a specific embodiment of the present invention include a decentralized node operation counter and a centralized hot and cold node compression pool, which provide historical access information and candidate evolution node information for node evolution. During indexing, the worker thread is responsible for updating the operation information of relevant nodes. Write operations trigger foreground node adjustments, and scan operations trigger background node compression. The background thread then selects and compresses qualified nodes from the hot and cold node compression pool.

[0098] See also Figure 4 The figure shows a schematic diagram of the hot and cold node compression pools of an embodiment of the present invention. The hot and cold node compression pools are composed of a hot scan sparse node pool, a cold access sparse node pool, and a node mapping map shared by the hot and cold node pools, and each NUMA node maintains a hot and cold node compression pool. Specifically, the hot scan sparse node pool stores nodes that are frequently accessed and sparsely structured on the current NUMA node, while the cold access sparse node pool stores nodes that are sparsely structured and have very low access frequency. Among them, the node pool is implemented by a bidirectional linked list. When adding a new linked list node to the hot scan sparse node pool and the cold access sparse node pool, it is added from the tail node. The closer the node in the cold access sparse node pool is to the head node, the less frequently the node is accessed. Therefore, the background thread starts scanning from the head node. Each time a node is compressed, the background thread will remove it from the compression pool.

[0099] Specifically, during the indexing phase, each cold access sparse node pool is initially empty. When the data node space overhead utilization is lower than the preset threshold, it is added to the cold access sparse node pool on the corresponding NUMA node. When the worker thread performs foreground adjustment, the old node is removed from the hot and cold node compression pool, and the node that meets the conditions is written to the cold access sparse node pool. In addition, before the background thread performs node compression, the relevant linked list nodes are also removed from the cold access sparse node pool.

[0100] Specifically, the hot scan sparse node pool is updated when a worker thread detects frequent scans on a leaf node. When a foreground worker thread performs a range scan on a segmented node or a sparse ordered gap node and detects that the scan operation increment on the current node has reached a threshold, the thread further calculates the sum of the scan operation increments since the last record, as well as the sum of all operation increments since the last record. If the scan operation ratio exceeds a threshold (default 0.5), the thread attempts to add the current node to the hot scan sparse node pool of the corresponding hot and cold node compaction pool.

[0101] Figure 5 FIG2 is a schematic diagram of foreground adjustment of the adaptive node evolution mechanism according to an embodiment of the present invention. The foreground adjustment process includes the steps of triggering node evolution, calculating new node layout, and performing node reconstruction.

[0102] Specifically, the conditions for triggering node evolution are divided into 1) segmented node write operations detect that the data in the local segment involved in the write reaches the preset upper limit; 2) ordered gap node write operations detect that the model fails (that is, the historical operation cost far exceeds the expected cost) or the amount of data in the node reaches the preset upper limit.

[0103] Specifically, computing node layout involves two steps: 1) centralizing the data in the nodes to be adjusted; 2) calculating the lowest-cost layout for operations weighted by historical operational load using a hybrid node cost model combined with a fan-out decision tree, such as the number of new nodes, data partitioning, and the type of new nodes. The fan-out decision tree enumerates a binary tree from 1 to a preset maximum fan-out based on the data range of the original node. When the data size of the enumerated data partitions meets the maximum data node size constraint, the weighted operation costs of the sequential gap nodes and segmented nodes under the linear model predictive control (LMPC) are calculated using the least squares method, and the fan-out partition with the lowest cumulative cost is selected.

[0104] Specifically, the hybrid node operation cost model is approximated in the following way: in ordered gap nodes, the query search cost is estimated based on the error between the actual position of the key and the model predicted position, while the insertion operation cost also includes the cost of inserting data movement on the basis of the query search, and is approximated by the sum of one-quarter of the square of the length of each continuous dense part of the array in the node; the query search cost and insertion cost of the segmented node are obtained by adding the costs of each segment; the cost calculation of the segment in the ordered phase is consistent with the gap array calculation method, and the cost in the semi-ordered phase is estimated based on the average level bucket data volume, where the average level bucket query search cost is estimated based on the range of the level bucket binary search, and the insertion cost is approximated by one-quarter of the average continuous dense array length in the level bucket; the comprehensive operation cost of the node is obtained by multiplying the query cost by the memory access coefficient and the insertion operation cost multiplied by the continuous memory access coefficient, weighted by the operation proportion; when the proportion of write operations in the node's historical workload exceeds a certain threshold, the cost of writing to the ordered gap node will be multiplied by a penalty factor due to the potential write concurrency conflicts and memory access congestion caused by write amplification in the ordered gap node.

[0105] Specifically, in the execution node reconstruction step selected by the foreground, the foreground thread directly executes node reconstruction under the requirements of the new node layout. If the number of new nodes is greater than 1, the leaf nodes of the relevant type are obtained according to the divided data. If the number of split nodes exceeds the repetition cardinality of the original node, the expansion of the parent node is triggered.

[0106] Figure 6 The figure shows a background adjustment diagram of the adaptive node evolution mechanism of an embodiment of the present invention. The background thread preferentially scans the hot scan sparse node pool from beginning to end. After reconfirming the validity of the node (such as frequent scanning but few insertions and poor spatial coherence), the non-range-friendly nodes that frequently perform range scans are converted into range-friendly nodes. Specifically, from the perspective of node type, the background thread converts sparse segmented data nodes into compact ordered gap nodes. When the background thread scans the cold access sparse node pool, the access coefficient of the current node is calculated (the proportion of the node's cumulative operations since the last record and the global cumulative operations in the same period). If the access coefficient is less than the inverse of the number of leaf nodes, the node is considered to be a cold access node. The space utilization of the node is further checked. The node will be compressed only when it meets the conditions of being a cold access node and low space utilization. Otherwise, the node will be moved to the tail of the cold access sparse node pool.

[0107] The concurrency management strategy for multi-threaded operations in the embodiment of the present invention is as follows:

[0108] Since writing to an ordered gap node involves moving data across the entire array at most, this node adopts a mutually exclusive concurrency strategy of optimistic reads and pessimistic writes.

[0109] Specifically, before performing a specific read operation on an ordered gap node, the node's basic concurrency control lock is first read. If the current lock is in a write-exclusive state, the query proceeds to step 1; otherwise, the current lock status version information is recorded. After obtaining the query result, before terminating, the current node's basic concurrency control lock status information is checked again. If it does not match the recorded information, the query proceeds to step 1; otherwise, the query ends.

[0110] Specifically, before executing a specific write operation in an ordered gap node, first attempt to obtain the node's basic concurrency control lock. If the acquisition fails, go to step 1 of the insertion. After writing the data, if the node evolution and reconstruction are not triggered, release the basic concurrency control lock and then end the insertion. If reconstruction is triggered, the basic concurrency control lock is not released.

[0111] Segment nodes employ finer-grained concurrency control because query and insert operations only involve a single segment. Each segment's metadata maintains a basic concurrency control lock. Normal operations between segments (those that do not trigger node reconstruction) are concurrent. The concurrency control logic for operations within a segment is similar to that of ordered gap nodes, with optimistic reads and pessimistic writes mutually exclusive.

[0112] Specifically, in a segment node, after the query operation locates a segment, the concurrent lock status of the segment metadata is accessed. If the current segment is in write-exclusive state, the query proceeds to step 1; otherwise, the current lock status version information is recorded. After obtaining the query result, before terminating, the basic concurrent control lock status information of the current segment is checked again. If it does not match the recorded information, the query proceeds to step 1; otherwise, the query ends.

[0113] Specifically, in a segment node, when the insert operation locates a segment, it first attempts to obtain the basic concurrency control lock of the segment metadata. If the acquisition fails, it goes to step 1 of the insert. After writing the data, if the node evolution and reconstruction is not triggered, the basic concurrency control lock is released and the insertion ends. If reconstruction is triggered, the basic concurrency control lock is not released.

[0114] In addition, in order to prevent background node adjustments from affecting foreground query operations, a node evolution lock based on a read-write lock is added to the leaf node. Foreground query operations only check the basic concurrent lock and are not affected by the node evolution lock. When ordinary write operations enter the node, they need to obtain the read lock of the node evolution lock and release the read lock of the node evolution lock after leaving. When the thread performs a structural reconstruction operation, it needs to obtain the write lock of the node evolution lock. If it was originally a read lock, it will be upgraded to a write lock.

[0115] In order to fully utilize the characteristics of the NUMA architecture, that is, each NUMA node contains one or more processors and local memory directly connected through a memory controller, the processor has lower latency and higher bandwidth when accessing local memory, and higher latency and lower bandwidth when accessing remote memory. The NUMA-aware memory management and task scheduling framework in the embodiment of the present invention adopts a weighted polling method based on the distribution of available processor computing resources on the NUMA node and a query redistribution scheduling mechanism. The former applies for memory from the memory space of each NUMA node and allocates it to each data node; the latter selectively schedules and redistributes query task requests to threads running on the processors of each NUMA node.

[0116] Specifically, the weighted polling method based on node resource distribution is used in the index batch loading establishment phase to allocate data layer nodes to each NUMA node. Assuming that the future thread resource distribution ratio on each node is r i , i=1…n, n is the number of nodes, the total proportion is 1, the node with the smallest resource proportion is taken as the baseline, and the coefficient a of each node relative to the baseline is obtained i , considering the locality of data, the framework uses the data nodes of the data layer as the division unit and the amount of data on the data nodes as the allocation indicator. Assuming that the average range scan size in the future workload is N rq , and the maximum amount of data that can be stored in the data leaf node is N m , take the minimum value of the two as the reference allocation unit value D r , in the batch loading phase, calculate the data size D of the accumulated data nodes on the current NUMA node x , if D x Greater than or equal to a i *D r *0.8, then its subsequent data nodes will rotate to the next NUMA node for allocation.

[0117] Specifically, the query redistribution scheduling mechanism first assumes that there are n NUMA nodes and the number of thread resources allocated to each NUMA node is t i , i=1…n, each thread has n chain arrays of queries to be reallocated, and is tied to a fixed NUMA node. Each thread is initially assigned a series of requests. During execution, each thread places some requests in the array to be reallocated according to a certain probability. After the thread finishes executing the first batch of data, it will be allocated 1 / t iThe ratio of the thread to be reallocated is obtained from the corresponding array to be reallocated of the thread located on other NUMA nodes and executed again. During the thread execution process, non-range operations are directly executed on the current thread. For range query operations involving a range close to the preset Dr, after the search layer is executed to locate the leaf node, if the NUMA node actually allocated to the leaf node is inconsistent with the current thread, that is, the execution and data coordination cannot be achieved, then the difference in the thread computing resource ratio delta between the current node and the actual node is calculated. x , if the random number generated between (0,1) is greater than delta x , then this operation will be redistributed and scheduled, directly returning the leaf node address and its actual node ID. Otherwise, the current operation will continue. When the thread executes a secondary allocation request, it will first try to reuse the address provided by the previous execution. If the address has not been marked as deleted, it will reuse the address. Otherwise, it will re-execute the complete search steps from the beginning.

[0118] Specifically, the steps for processing a point query in the embodiment of the present invention are as follows:

[0119] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its model until reaching the leaf node, that is, the data node;

[0120] Step 2: If the data node type is an ordered gap node, use the linear model within the node to make a prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends;

[0121] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5;

[0122] Step 4: Based on the initial position predicted by the model, the segment size is modulo-ed to obtain the intra-segment offset (if a metadata slot is encountered, the offset is shifted right by one position). If the data at this offset does not match the query, an exponential search is used to correct the prediction error within the segment to obtain the query result, and the query ends.

[0123] Step 5: Based on the query keyword hash, the corresponding offset in the segment is obtained (the offset position is shifted right by one when encountering a metadata slot), and the hierarchical bucket status at the current offset is determined based on the slot status bitmap in the segment metadata. If the hierarchical bucket is in a virtual state, the key stored at the segment offset is directly compared and the matching result is returned. Otherwise, the hierarchical bucket status is further determined based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, the buffer array is directly scanned for matching and the matching result is returned. Otherwise, the hierarchical bucket structure is a hierarchical base bucket, and the process proceeds to step 6.

[0124] Step 6: First, read the metadata of the hierarchical base bucket to obtain the bucket level number and the first address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group of the bottom layer array, obtain the query result, and the query ends.

[0125] Specifically, the processing point insertion steps of the embodiment of the present invention are as follows:

[0126] Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of its model until reaching the data node;

[0127] Step 2: If the data node type is an ordered gap node, use the intra-node linear model for prediction and use exponential search to query the insertion position in the node array as needed. Perform move insertion or deletion, update local insertion and deletion statistics, and check the node status. If the current node model fails or the data volume triggers the threshold, go to step 6; otherwise, the insertion ends.

[0128] Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5;

[0129] Step 4: Based on the initial position predicted by the model, the segment size is modulo-calculated to obtain the intra-segment offset. If the data at this offset does not meet the data order requirement, an exponential search is used to correct the prediction error within the segment to obtain the insertion position. The nearest gap is found through the metadata bitmap to perform the move insertion. If the current stage data volume reaches the preset density limit, the segment structure changes from the ordered stage to the semi-ordered stage, and the data is redistributed. Insertion is completed;

[0130] Step 5: Based on the inserted keyword hash, the corresponding offset in the segment is obtained, and the state of the hierarchical bucket at the current offset is determined according to the data slot state bitmap in the metadata. If the hierarchical bucket is in a virtual state, it is directly inserted. If a conflict occurs, a simple buffer array is requested as the hierarchical bucket, and the insertion is completed. If the hierarchical bucket is a simple buffer array, simple moving insertion is performed. If the buffer array density reaches the preset value, it is converted to a hierarchical reference bucket, and the insertion is completed. If the hierarchical bucket is a hierarchical reference bucket, the metadata of the hierarchical reference bucket is read, a binary search is performed in the reference layer to locate the specific group of the underlying array, and a moving insertion is performed. If there is no gap in the group, the nearest gap is found from the adjacent group, the moving insertion is performed, and the benchmark value of the corresponding position in the benchmark layer is updated. If it is found that the hierarchical reference bucket reaches the preset density upper limit after the insertion, go to step 6, otherwise the insertion is completed.

[0131] Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information combined with data distribution and historical operation statistics. After that, mark the original node as deleted and the insertion is completed.

Claims

1. A NUMA architecture-aware, highly scalable learning index construction method, characterized in that: include: Constructing a hybrid node tree index structure, the structure including internal nodes and leaf nodes, wherein the internal nodes and leaf nodes respectively occupy physical areas of a memory; The construction of the internal nodes includes: configuring a linear model, establishing a main array for storing pointers to addresses of nodes in the next layer; The leaf nodes are divided into ordered gap nodes and segmented nodes; the construction of the ordered gap nodes includes: configuring a linear model, establishing a gap array storage structure, arranging the data in the gap array according to the model prediction position, and setting a bitmap array to mark the gap state, and the gap is filled with adjacent keywords; The construction of the segment node includes: configuring a linear model, dividing the main array into logical segments of equal length, each segment contains a metadata part and a data part, the metadata part records the segment status information, including data volume, segment stage status, concurrency control status and intra-segment slot status bitmap, and the data part stores actual data.

2. The method according to claim 1, characterized in that The construction of the segment node also includes: Set the ordered and semi-ordered stages of the segmentation; In the ordered phase, data are arranged in a gap array manner; In the semi-order phase, data is hashed into hierarchical buckets; The construction of the hierarchical bucket includes: Set the virtual state so that the data is directly stored in the main array when there is no conflict; Set the simple buffer array state and allocate the buffer array when the conflict is less than the preset value; Set the hierarchical base bucket status and allocate the hierarchical base bucket when the number of conflicts exceeds the preset value; The construction of the hierarchical benchmark bucket includes: Construct multi-layer ordered compact arrays; Set up to three layers of expansion mechanism; Configure the baseline layer to store only keywords; Ensure that each layer of the structure meets cache line alignment requirements.

3. A NUMA architecture-aware highly scalable learning indexing system, characterized by: The system adopts a hybrid node tree index structure, which includes internal nodes and leaf nodes, wherein the internal nodes and leaf nodes occupy physical areas of the memory respectively; The internal nodes contain a linear model and a main array storing node pointers, and accurate search is achieved through the linear model; the leaf nodes are divided into ordered gap nodes and segment nodes, the ordered gap nodes use a gap array structure to store data, and the segment nodes use a logical segment structure to store data; The indexing system implements an adaptive node evolution mechanism based on an operation cost model; The indexing system adopts a hybrid node operation cost model; The indexing system implements a multi-threaded concurrency control strategy; The system includes a NUMA-aware memory management and thread scheduling framework.

4. The NUMA architecture-aware highly scalable learning indexing system according to claim 3, characterized in that: The adaptive node evolution mechanism includes foreground adjustment and background adjustment; The foreground adjustment is performed by the foreground write operation thread, which reduces the operation cost by triggering node evolution, calculating the new node layout and performing node reconstruction; The background adjustment is performed by a background thread allocated on each NUMA node to periodically scan the hot and cold node compression pools on the corresponding NUMA node and compress the nodes to optimize the performance of range operations and the space overhead of indexes; The new node layout calculation in the foreground adjustment is based on a decentralized node operation counter and is obtained through a fan-out layout decision tree evaluation in combination with a hybrid node operation cost model; The decentralized node operation counter is used to record the operation increment and operation history information of each thread in the leaf node, including the cumulative number of global operations recorded last time and the number of data nodes in the index, the cumulative number of each type of operation history, and the operation increment count under each thread; The centralized hot and cold node compression pool includes a hot scan sparse node pool, a cold access sparse node pool, and a node mapping map shared by the hot and cold node pools, wherein the hot scan sparse node pool stores sparse nodes to be optimized whose range scan operation times are higher than a specified threshold, and the cold access sparse node pool stores sparse nodes to be optimized whose access frequency is lower than a specified threshold.

5. The NUMA architecture-aware highly scalable learning indexing system according to claim 3, characterized in that The hybrid node operation cost model includes: In ordered gap nodes, the cost of query search is estimated based on the error between the actual position of the key and the model's predicted position, while the cost of insert operation also includes the cost of inserting data movement on top of the query search, which is approximated by the sum of one-quarter of the square of the length of each continuous dense part of the array in the node; The query search cost and insertion cost of a segment node are obtained by adding the costs of each segment; The cost calculation method for the segmentation in the ordered phase is the same as that for the gap array. The cost in the semi-ordered phase is estimated based on the average amount of data in the level bucket. The average query search cost for the level bucket is estimated based on the range of the binary search for the level bucket. The insertion cost is approximated by one-quarter of the average length of the continuous dense array in the level bucket. The comprehensive operation cost of a node is obtained by multiplying the query cost by the memory access coefficient and the insertion operation cost by the continuous memory access coefficient, weighted by the operation ratio; When the proportion of write operations in the node's historical workload exceeds a specified threshold, the cost of writing to the ordered gap node will be multiplied by a penalty factor to reduce potential write concurrency conflicts on the ordered gap node and memory access congestion caused by write amplification.

6. The NUMA architecture-aware highly scalable learning indexing system according to claim 3, characterized in that: The multi-threaded concurrency control strategy includes: Leaf nodes use basic concurrency locks and node evolution locks based on read-write locks to manage concurrency between non-structural reconstruction operations and concurrency between non-structural reconstruction operations and structural reconstruction operations respectively; The basic concurrent lock allows concurrent read operations through optimistic version control, and the mutual exclusion lock design only allows one thread to acquire the lock, that is, read operations are concurrent, write operations are mutually exclusive, and read and write operations are mutually exclusive; Ordered gap nodes contain a basic concurrency lock, while segmented nodes maintain a basic concurrency lock in the metadata of each segment. Therefore, ordered gap nodes only allow one thread to perform write operations at the same time, while segmented nodes allow multiple threads to perform write operations on multiple segments within the node at the same time. The node evolution lock based on the read-write lock manages non-structural reconstruction write operations in the form of a read lock and manages structural reconstruction operations in the form of a write lock.

7. The NUMA architecture-aware highly scalable learning indexing system according to claim 3, characterized in that: The NUMA-aware memory management and thread scheduling framework includes: NUMA-aware memory management uses a weighted round-robin method based on node resource distribution to calculate the data size of the accumulated data nodes on the NUMA node, allocate the next data node to the corresponding NUMA node according to the relative baseline coefficient, and allocate the data nodes to each NUMA node; The NUMA-aware query reallocation scheduling mechanism dynamically determines the reallocation of query requests based on the difference in computing resource ratio between the NUMA node where the actual leaf node is located and the node where the current thread executing the request is located.

8. The point query operation method of the NUMA architecture-aware highly scalable learning indexing system according to any one of claims 3 to 7, comprising the following steps: Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of their linear models until reaching the data node; Step 2: If the data node type is an ordered gap node, use the linear model within the node to make a prediction, use exponential search to correct the prediction error within the node array, obtain the query result, and the query ends; Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5; Step 4: The initial position predicted by the linear model is modulo the segment size to obtain the intra-segment offset. If the offset data does not match the query, the exponential search is used to correct the prediction error within the segment to obtain the query result, and the query ends. Step 5: Obtain the corresponding offset within the segment based on the query keyword hash, and determine the hierarchical bucket status at the current offset based on the slot status bitmap in the segment meta-information. If the hierarchical bucket is in a virtual state, directly compare it with the key stored at the segment offset and return the result based on the match. Otherwise, further determine the hierarchical bucket status based on the highest bit of the address stored at the segment offset. If the hierarchical bucket is a simple buffer array, directly scan the buffer array for a match and return the result based on the match. Otherwise, the hierarchical bucket structure is a hierarchical base bucket, and proceed to step 6. Step 6: Read the metadata of the hierarchical base bucket to obtain the bucket layer number and the first address of the corresponding layer array. Perform a binary search in the base layer to locate the specific group of the bottom layer array, obtain the query result, and the query ends.

9. The point insertion operation method of the NUMA architecture-aware highly scalable learning indexing system according to any one of claims 3 to 7, comprising the following steps: Step 1: Starting from the root node, traverse the internal nodes and determine the next layer of nodes to be visited based on the accurate prediction results of their linear models until reaching the data node; Step 2: If the data node type is an ordered gap node, use the intra-node linear model for prediction and use exponential search to query the insertion position in the node array as needed. Perform move insertion or deletion, update local insertion and deletion statistics, and check the node status. If the current node linear model fails or the data volume triggers the threshold, go to step 6; otherwise, the insertion ends. Step 3: If the data node type is a segmented node, first use the linear model within the node to predict the initial position, calculate the corresponding segment ID according to the integer multiple of the segment size corresponding to the position, and read the segment metadata. If the segment is in the ordered stage, go to step 4; if the segment is in the semi-ordered stage, go to step 5; Step 4: The initial position predicted by the linear model is modulo the segment size to obtain the intra-segment offset. If the data at this offset cannot meet the data order requirement, an exponential search is used to correct the prediction error within the segment to obtain the insertion position. The nearest gap is found through the metadata bitmap to perform the move insertion. If the current stage data volume reaches the preset density limit, the segment structure changes from the ordered stage to the semi-ordered stage, the data is redistributed, and the insertion ends. Step 5: Based on the inserted keyword hash, the corresponding offset in the segment is obtained, and the state of the hierarchical bucket at the current offset is determined according to the data slot state bitmap in the metadata. If the hierarchical bucket is in a virtual state, it is directly inserted. If a conflict occurs, a simple buffer array is requested as the hierarchical bucket, and the insertion is completed. If the hierarchical bucket is a simple buffer array, simple moving insertion is performed. If the buffer array density reaches the preset value, it is converted to a hierarchical reference bucket, and the insertion is completed. If the hierarchical bucket is a hierarchical reference bucket, the metadata of the hierarchical reference bucket is read, a binary search is performed in the reference layer to locate the specific group of the underlying array, and a moving insertion is performed. If there is no gap in the group, the nearest gap is found from the adjacent group, the moving insertion is performed, and the benchmark value of the corresponding position in the benchmark layer is updated. If it is found that the hierarchical reference bucket reaches the preset density upper limit after the insertion, go to step 6, otherwise the insertion is completed. Step 6: Based on the insertion feedback, collect the valid data in the current data node, and perform node evolution reconstruction based on the feedback information combined with data distribution and historical operation statistics, and the insertion is completed.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the point query operation method or the point insertion operation method of the NUMA architecture-aware highly scalable learning indexing system as described in claim 8 or 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the point query operation method or the point insertion operation method of the NUMA architecture-aware highly scalable learning indexing system as described in claim 8 or 9 are implemented.

Images