A learning index method and system supporting high concurrency scenarios

Abstract

The application discloses a kind of learning index systems of supporting high concurrency scene, including receiving request from user, and according to the interface that user sends the request judges the request is single point query request, range query request or write request, if it is write request, according to the data point to be inserted obtained by the write request, and determine the leaf node corresponding to the data point in the recursive model index established in advance, according to the data point to be inserted obtained by the second module generates an incremental record, and the incremental record is inserted into the incremental linked list in the corresponding incremental buffer area, and whether the number of incremental records stored in incremental linked list reaches the preset threshold value is judged.The application can solve the technical problem that the learning index method using the tree-based incremental buffer area and the learning index method using the gap array both use fine-grained lock and optimistic concurrency control technology to realize the concurrent access of index, resulting in expensive lock overhead.

Description

Technical Field

[0001] This invention belongs to the field of computer data storage, and more specifically, relates to a learning indexing method and system that supports high-concurrency scenarios. Background Technology

[0002] The explosive growth of data poses a severe challenge to data management and efficient retrieval in storage systems. However, traditional index structures have limitations in terms of performance and space efficiency, failing to meet the demands for efficient data access in concurrent scenarios. In recent years, machine learning has made continuous progress, and learning indexes have pioneered a new direction for the application of machine learning in index optimization. The core idea of ​​learning indexes is that existing index structures can be replaced by other types of models. These models can learn the distribution of keys, and using this information, they can effectively predict the location or existence of records.

[0003] Existing learning indexing methods mainly include the following: The first method uses a tree-based incremental buffer to handle insertion operations. This method mainly utilizes a tree-based index structure (such as B+ tree, Masstree, etc.) as an incremental buffer to receive newly inserted data, and periodically merges the incremental buffer with leaf nodes to generate a new machine learning model. The second method uses a gap array (GA) to handle insertion operations. This method mainly reserves gaps in leaf nodes to receive newly inserted data. When the target position is empty, data can be inserted directly. If the target position is not empty, the data between the target position and the nearest empty position needs to be moved to make room for the new data. In addition, to support concurrent access, both of the above learning indexing methods use fine-grained locking and optimistic concurrency control techniques to handle thread conflicts.

[0004] However, the aforementioned existing learning indexing methods all have some significant drawbacks:

[0005] First, both the tree-based incremental buffer learning indexing method and the gap array-based learning indexing method employ fine-grained locking and optimistic concurrency control techniques to achieve concurrent access to the index. As the number of concurrent accesses increases, this can lead to expensive locking overhead. In addition, fine-grained locking may also cause thread safety issues.

[0006] Second, the tree-based incremental buffer learning indexing method will generate high structure modification overhead. As the amount of data in the incremental buffer increases, due to the serious data dependencies between nodes, structure modification operations (SMO) need to be performed frequently, which will consume a lot of time and block other write threads, reducing concurrency performance.

[0007] Third, the learning indexing method using tree-based incremental buffers still uses traditional binary search to perform searches in the incremental buffer, which cannot benefit from the machine learning model, resulting in poor search and insertion performance.

[0008] Fourth, both the tree-based incremental buffer learning indexing method and the gap array-based learning indexing method directly modify the index structure, which can lead to CPU cache invalidation and reduced concurrency performance. Summary of the Invention

[0009] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a learning indexing method and system that supports high-concurrency scenarios. Its purpose is to solve the technical problems arising from the high lock overhead and thread safety issues associated with existing learning indexing methods using tree-based incremental buffers and gap array-based learning buffers, which employ fine-grained locking and optimistic concurrency control techniques to achieve concurrent access to the index. These problems include: the high lock overhead caused by fine-grained locking; the time-consuming and time-consuming nature of existing tree-based incremental buffer learning indexing methods due to frequent structure modification operations, which also blocks other write threads and reduces concurrency performance; the poor search and insertion performance caused by using traditional binary search to perform searches in the incremental buffer; and the CPU cache invalidation and reduced concurrency performance caused by direct modifications to the index structure in both existing tree-based incremental buffer and gap array-based learning indexing methods.

[0010] To achieve the above objectives, according to one aspect of the present invention, a learning indexing method supporting high-concurrency scenarios is provided, comprising the following steps:

[0011] (1) Receive a request from the user and determine whether the request is a single-point query request, a range query request, or a write request based on the interface through which the user sent the request. If it is a single-point query request, proceed to step (4); if it is a range query request, proceed to step (7); if it is a write request, proceed to step (2).

[0012] (2) Obtain the data point to be inserted according to the write request, and determine the leaf node corresponding to the data point in the pre-established recursive model index.

[0013] (3) Generate an incremental record based on the data point to be inserted obtained in step (2), insert the incremental record into the incremental linked list in the corresponding incremental buffer, and determine whether the number of incremental records stored in the incremental linked list has reached the preset threshold. If so, retrain the incremental linked list and send a notification of successful write operation to the user. The process ends. Otherwise, send a notification of successful write operation to the user. The process ends.

[0014] (4) Obtain the data point of the single-point query request and the incremental buffer where the single-point query request is located, and use the Bloom filter to determine whether the corresponding result of the single-point query request can be found in the incremental buffer. If yes, proceed to step (5); otherwise, return an empty result to the user and the process ends.

[0015] (5) Determine whether the result corresponding to the single-point query request can be found in the incremental linked list in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, proceed to step (6).

[0016] (6) Determine whether the result corresponding to the single-point query request can be found in the incremental node in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, return an empty result to the user and the process ends.

[0017] (7) Obtain the starting value and ending value corresponding to the range query request and the leaf node where the starting value of the range query request is located. Determine the starting position of the starting value among all data points managed by the leaf node. Starting from the starting position, traverse all data points managed by the leaf node whose keys are between the starting value and the ending value. Store all data points in the result set R1 and perform further search in the incremental linked list and incremental node in the incremental buffer associated with the starting position to obtain all data points whose keys are between the starting value and the ending value. Then proceed to step (8).

[0018] (8) Obtain all incremental linked lists in the incremental buffer obtained in step (7), traverse all incremental linked lists, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R2. Merge the result set R2 with R1 to obtain a new result set R3.

[0019] (9) Obtain all incremental nodes in the incremental buffer obtained in step (7), traverse all incremental nodes, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R4. Merge the result set R4 with R3 to obtain a new result set R5.

[0020] Preferably, in step (1), the specific type of the request is determined by checking the interface through which the user sends the request. If it is a single-point query request, the user sends the request through the single-point query interface. If it is a range query request, the user sends the request through the range query interface. If it is a write request, the user sends the request through the write program interface.

[0021] Step (2) is to query the leaf node corresponding to the data point in the recursive model index. Specifically, the process is as follows: First, the binary search method is used to determine the real position of the key of the data point in the recursive model index, and the leaf node at the real position is obtained as the further search object.

[0022] Preferably, the recursive model index in step (2) is established according to the following steps:

[0023] (A1) Obtain the dataset keys = {(key a ,pos a ), where a = 1, 2, ..., n}, where n represents the total number of data points in the dataset, i ∈ [1, n], key i pos represents the key of the i-th data point. i This indicates the position of the i-th data point in the dataset keys;

[0024] (A2) Set counter j = 1 and counter i = 2, and set the first data point (key1, pos1) in the dataset keys as the current starting point of leaf node j, and obtain point S according to the pre-set error threshold δ. a = (key1,pos1+δ) and point S b = (key1, pos1―δ); where the error threshold δ is any natural number. The smaller the value, the more leaf nodes are generated, which will affect the performance of the root node searching for leaf nodes. The larger the value, the larger the error of the linear regression model trained based on the data points managed by the leaf nodes.

[0025] (A3) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8). Otherwise, generate point S based on the error threshold δ and the second data point (key2, pos2) in the dataset keys. c = (key2, pos2+δ) and point S d =(key2,pos2―δ), obtain point S a and point S d The generated line segment l1 and its slope slope1 are used to obtain point S. b and point S cGenerate line segment l2 and its slope slope2, and obtain the intersection point S0(key0,pos0) of line segment l1 and line segment l2. Set the counter i = i + 1, and then proceed to step (A4).

[0026] (A4) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8). Otherwise, generate point S based on the error threshold δ and the i-th data point in the dataset keys. e =(key i ,pos i +δ) and point S f =(key i ,pos i ―δ), set counter i = i + 1, and calculate point S e The slope S between point S0 and point S0 high and point S f The slope S between point S0 and point S0 low And determine whether there is an S high Slope2 ≤ Slope1 ≤ S low If yes, proceed to step (A5); otherwise, proceed to step (A8).

[0027] (A5) Determine if |pos exists. i ―slope1(key i If key0)pos0|>δ, then update. Then proceed to step (A6); otherwise, proceed to step (A6).

[0028] (A6) Determine if |pos exists. i ―slope2(key i If key0)pos0|>δ, then update. Then proceed to step (A7); otherwise, proceed to step (A7).

[0029] (A7) Set the counter i = i + 1 and return to step (A4);

[0030] (A8) Data point (key) i―1 ,pos i―1 All data points up to the current starting point are treated as data managed by the j-th leaf node, and it is determined whether the total number of all data points is greater than 1. If so, the slope of the linear regression model is set. Set the intercept of the linear regression model to intercept = pos0 - slope * key0, set the counter j = j + 1, and then proceed to step (A9). Otherwise, set the slope of the linear regression model to slope = 0 and the intercept to intercept = pos0. i―1Set the counter j = j + 1, and then proceed to step (A9);

[0031] (A9) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A10); otherwise, remove the data point (key). i ,pos i Set the current starting point of leaf node j, and generate point S according to the error threshold. a = (key1,pos1+δ) and point S b = (key1, pos1―δ), then return to step (A3);

[0032] (A10) Select the starting point of each leaf node and the leaf node (i.e., ...<MinKey,Data Node> This is stored as metadata in the root node, thus creating the recursive model index.

[0033] Preferably, incremental records are represented by triples of <write operation type, key, value>;

[0034] The process of establishing the incremental buffer is as follows: an incremental buffer is allocated for each data point. The incremental buffer consists of four parts, including an incremental list, incremental nodes, a Bloom filter, and a structure adjuster. The incremental list and incremental nodes are initially empty.

[0035] Preferably, the process of inserting the incremental record into the incremental linked list in the incremental buffer in step (3) is as follows: first, the incremental record is pointed to the head of the incremental linked list by the linked list pointer, and then the incremental record is inserted into the head of the incremental linked list by the CAS atomic instruction to complete the insertion operation.

[0036] Preferably, the retraining of the incremental linked list in step (3) involves merging the data points in the incremental linked list in the incremental buffer with the data points in the incremental nodes of the incremental buffer. This process includes the following sub-steps:

[0037] (3-1) Create two empty sets S insert and S delete S insert S is used to store the keys in incremental records where the write operation type is insert. delete Used to store keys in incremental records where the write operation type is delete;

[0038] (3-2) Scan the incremental linked list sequentially. Whenever an incremental record with key k and write operation type insertion is encountered, and k is not in set S... delete In the middle, k is added to set S. insert Meanwhile, whenever an incremental record with key value k and write operation type delete is scanned, and k is not in set S...insert In the middle, k is added to set S. delete In the case of k, k is a natural number;

[0039] (3-3) Set S insert and set S delete The merge is performed, and the merged results are sorted by key. Then, the sorted key-value pairs are merged with the key-value pairs in the incremental node to generate a new incremental node P containing all data points.

[0040] (3-4) Replace the incremental node with the new incremental node P obtained in step (3-3) using the CAS atomic instruction;

[0041] (3-5) Determine whether the number of data points in the new incremental node obtained in step (3-4) is greater than or equal to the preset splitting threshold (in this embodiment, the value is equal to 512). If so, split the incremental node to obtain two new incremental nodes, and then proceed to step (3-7); otherwise, proceed to step (3-6).

[0042] (3-6) Determine whether the number of data points in the new incremental node obtained in step (3-4) is lower than the preset merging threshold (in this embodiment, the value is equal to 32). If so, merge the incremental node P with the sibling node located to the right of the incremental node in the recursive model index to obtain a new incremental node, and then proceed to step (3-12). Otherwise, the process ends.

[0043] (3-7) Divide the data points in the incremental node P obtained in step (3-4) equally, select the key b corresponding to the middle data point as the separator key, and create a new incremental buffer. The incremental buffer contains an incremental node Q. The incremental node P stores all the data points in the incremental node P whose keys are greater than or equal to the separator key b. Then proceed to step (3-8).

[0044] (3-8) Generate an incremental record Δsplit and set the incremental node Q as the sibling node to the right of the incremental record Δsplit in the recursive model index. Then insert the incremental record Δsplit into the incremental node P and determine whether other incremental records have been inserted into the incremental node P before the insertion of the incremental record Δsplit. If so, it means that the CAS atomic instruction failed to execute and then proceed to step (3-9). Otherwise, it means that the incremental record Δsplit was successfully inserted into the incremental node P and all data points in the incremental node P whose keys are greater than or equal to the separator key b in step (3-7) have become invalid. Proceed to step (3-10).

[0045] (3-9) Reclaim the incremental record Δsplit generated in step (3-8), divide the data points in the incremental node P obtained in step (3-4) into equal parts, select the key corresponding to the middle data point as the separator key, generate a new incremental record Δsplit, insert the incremental record Δsplit into the incremental node P, and then return to step (3-8).

[0046] (3-10) Insert an incremental record into the leaf node determined in step (2) (the purpose is to ensure that the incremental node Q can be directly searched), and then proceed to step (3-14).

[0047] (3-11) Obtain the sibling node R to the right of the incremental node P obtained in step (3-4) in the recursive model index, and generate an incremental record Δremove. Insert the incremental record Δremove into the incremental node R through the CAS atomic instruction.

[0048] (3-12) Generate an incremental record Δmerge, which contains a physical pointer to the incremental node R in step (3-12), and insert the incremental record Δmerge into the incremental node P obtained in step (3-4) using the CAS atomic instruction, and then proceed to step (3-13).

[0049] (3-13) Insert an incremental record into the leaf node obtained in step (2) (the purpose is to avoid searching for the incremental node R), and then proceed to step (3-14);

[0050] (3-14) Determine whether the number of data points in the incremental node P after the incremental record Δmerge is inserted in step (3-12) exceeds the preset threshold. If so, train the linear regression model based on the incremental node and then proceed to step (3-15). Otherwise, the process ends.

[0051] (3-15) Obtain all data points in the incremental node P after the incremental record Δmerge was inserted in step (3-12), and use all data points as the dataset to train the linear regression model;

[0052] Preferably, the merging method of step (3-3) is as follows;

[0053] S new =S insert ∪S old -S delete

[0054] Where S old and S new These represent the sets of key-value pairs in the incremental node and the new incremental node, respectively.

[0055] Preferably, the process of obtaining the incremental buffer where the single-point query request is located in step (4) is as follows: First, the root node in the recursive model index is searched using the binary search method to determine the leaf node corresponding to the single-point query request. Then, the predicted position of the data point of the single-point query request in all data points managed by the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using the binary search method, the true position of the data point of the single-point query request in the leaf node is determined. The incremental buffer associated with the true position is the incremental buffer where the single-point query request is located.

[0056] The process of querying the result corresponding to the single-point query request in the incremental node in step (6) is as follows: If the number of incremental data records in the incremental node is less than a preset threshold (preferably 256), the incremental node is searched using the binary search method to determine the true position of the data point of the single-point query request in the incremental node; otherwise, firstly, the predicted position of the key of the data point of the single-point query request in the incremental node is calculated using the linear regression model corresponding to the incremental node. Finally, based on the predicted position and the error range of the linear regression model, the true position of the data point corresponding to the single-point query request in the incremental node is determined using the binary search method, and the data point at the true position is obtained as the query result.

[0057] In step (7), the process of determining the starting position of the starting value among all data points is as follows: First, the root node in the recursive model index is searched using the binary search method to determine the leaf node corresponding to the starting value of the range query request. Then, the predicted position of the starting value of the range query request in the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using the binary search method to determine the true position of the starting value of the range query request in the leaf node, the true position is taken as the starting position, and the incremental buffer associated with the starting position is taken as the further search object.

[0058] According to another aspect of the present invention, a learning indexing system supporting high-concurrency scenarios is provided, comprising:

[0059] The first module is used to receive requests from users and determine whether the request is a single-point query request, a range query request, or a write request based on the interface through which the user sent the request. If it is a single-point query request, it proceeds to the fourth module; if it is a range query request, it proceeds to the seventh module; and if it is a write request, it proceeds to the second module.

[0060] The second module is used to obtain the data point to be inserted according to the write request, and to determine the leaf node corresponding to the data point in the pre-established recursive model index.

[0061] The third module is used to generate an incremental record based on the data point to be inserted obtained from the second module, insert the incremental record into the incremental linked list in the corresponding incremental buffer, and determine whether the number of incremental records stored in the incremental linked list has reached a preset threshold. If so, the incremental linked list is retrained and a notification of successful write operation is sent to the user, and the process ends. Otherwise, a notification of successful write operation is sent to the user, and the process ends.

[0062] The fourth module is used to obtain the data point of the single-point query request and the incremental buffer where the single-point query request is located, and to use a Bloom filter to determine whether the corresponding result of the single-point query request can be found in the incremental buffer. If so, proceed to the fifth module; otherwise, return an empty result to the user and the process ends.

[0063] The fifth module is used to determine whether the result corresponding to the single-point query request can be found in the incremental linked list in the incremental buffer obtained in the fourth module. If so, the result is returned to the user and the process ends; otherwise, the process proceeds to the sixth module.

[0064] The sixth module is used to determine whether the result corresponding to the single-point query request can be found in the incremental node in the incremental buffer obtained by the fourth module. If so, the result is returned to the user and the process ends; otherwise, an empty result is returned to the user and the process ends.

[0065] The seventh module is used to obtain the start and end values ​​corresponding to the range query request and the leaf node where the start value of the range query request is located. It determines the corresponding starting position of the start value among all data points managed by the leaf node. Starting from the starting position, it traverses all data points managed by the leaf node whose keys are between the start and end values, and stores all data points in the result set R1. It then performs further searches in the incremental linked list and incremental nodes in the incremental buffer associated with the starting position to obtain all data points whose keys are between the start and end values, and then proceeds to the eighth module.

[0066] The eighth module is used to obtain all incremental linked lists in the incremental buffer obtained by the seventh module, traverse all incremental linked lists, extract all data points whose keys are between the start value and the end value, and store all data points in the result set R2. The result set R2 is merged with R1 to obtain a new result set R3.

[0067] The ninth module is used to obtain all incremental nodes in the incremental buffer obtained by the seventh module, traverse all incremental nodes, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R4. The result set R4 is merged with R3 to obtain a new result set R5.

[0068] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0069] (1) Because the present invention adopts step (3), it implements a concurrent incremental write algorithm based on CAS atomic instructions, which can avoid the use of locks, store the modification in the incremental record, and insert the incremental record into the incremental buffer in an append write manner through CAS atomic instructions. Therefore, it can solve the technical problem that existing learning indexing methods using tree-based incremental buffers and learning indexing methods using gap arrays both use fine-grained locks and optimistic concurrency control technology to achieve concurrent access to the index, which will cause expensive lock overhead as the number of concurrent accesses increases.

[0070] (2) Because the present invention adopts step (3), it realizes the synchronization between threads through CAS atomic instructions and introduces the Help-along mutual assistance mechanism to require the threads to cooperate to complete the model retraining, which effectively avoids the thread safety problem caused by locks and improves the security of the index structure.

[0071] (3) Since the present invention adopts steps (3-1) to (3-14), it implements a non-blocking model retraining algorithm, which can perform incremental node splitting and merging operations in a non-blocking manner according to the changes in data distribution. Therefore, it can solve the technical problem that the existing learning indexing method using tree-based incremental buffers needs to frequently perform structural modification operations, resulting in a lot of time consumption and blocking other write threads, thus reducing concurrency performance.

[0072] (4) By adopting steps (3-15) and (4), this invention implements two accelerated search strategies in the incremental buffer. By using Bloom filters and machine learning models to accelerate the search of incremental linked lists and incremental nodes respectively, it can effectively reduce unnecessary memory access and improve the performance of indexing operations. This solves the technical problem that existing learning indexing methods using tree-based incremental buffers still use traditional binary search to perform searches in the incremental buffer, resulting in poor search and insertion performance.

[0073] (5) In this invention, step (3) is adopted, which inserts data into the incremental buffer in an incremental manner through the CAS atomic instruction, avoiding the CPU cache failure caused by directly changing the index structure, thereby improving the concurrent performance of the index structure. Finally, it solves the technical problem that the existing learning index method using tree-based incremental buffer and the learning index method using gap array both directly change the index structure, which will cause the CPU cache to fail and reduce the concurrent performance. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating the learning indexing method for high-concurrency scenarios supported by the present invention.

[0075] Figure 2 This is a schematic diagram of the CAS-based concurrent incremental write algorithm in the incremental buffer of this invention;

[0076] Figure 3 This is a schematic diagram of incremental linked list retraining in the incremental cache area of ​​this invention;

[0077] Figure 4 This is a schematic diagram of incremental node splitting in the incremental cache area of ​​the present invention;

[0078] Figure 5 This is a schematic diagram of incremental node merging in the incremental cache area of ​​the present invention;

[0079] Figure 6 These are the read load test results of this invention on the Normal dataset, Lognormal dataset, Facebook dataset, and OSM dataset;

[0080] Figure 7 These are the write load test results of this invention on the Normal dataset, Lognormal dataset, Facebook dataset, and OSM dataset. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0082] The technical terms used in this invention will be explained and described below:

[0083] Delta Buffer: Used to receive newly inserted data. When the number of incremental data records reaches a certain threshold, model retraining is required.

[0084] Delta Chain: A linked list structure in the incremental buffer that stores a series of incremental data, with the incremental data linked together by pointers. When the number of incremental data records reaches a certain threshold, the delta chain needs to be retrained, which involves merging the delta chain and the delta nodes to generate a new delta node containing all the incremental records.

[0085] Delta Node: A structure in the delta buffer that stores a certain number of delta data records. When the number of delta data records reaches a certain threshold, delta node retraining is required, which involves splitting and merging delta nodes.

[0086] Bloom filter: Used to quickly check if an element exists in a set, but has a certain false positive rate.

[0087] Structure Modification Operation (SMO): For tree structures, this refers to the operations of splitting and merging nodes.

[0088] Compare-and-Swap (CAS): An atomic instruction that compares the value VALUE stored at memory address ADDRESS with the expected result EXPECTED. If the two values ​​are equal, the new value NEW replaces the original value VALUE at memory address ADDRESS.

[0089] like Figure 1 As shown, this invention provides a learning indexing method that supports high-concurrency scenarios, comprising the following steps:

[0090] (1) Receive a request from the user and determine whether the request is a single-point query request, a range query request, or a write request based on the interface through which the user sent the request. If it is a single-point query request, proceed to step (4); if it is a range query request, proceed to step (7); if it is a write request, proceed to step (2).

[0091] Specifically, in this step, the specific type of request is determined by examining the interface through which the user sent the request. If it is a single-point query request, the user sent the request through the single-point query interface; if it is a range query request, the user sent the request through the range query interface; and if it is a write request, the user sent the request through the write application interface.

[0092] (2) Obtain the data point to be inserted according to the write request, and determine the leaf node corresponding to the data point in the pre-established recursive model index.

[0093] Specifically, the process of querying the leaf node corresponding to the data point in the recursive model index in this step is as follows: First, the binary search method is used to determine the real position of the key of the data point in the recursive model index, and the leaf node at the real position is obtained as the further search object.

[0094] The advantage of this step is that it avoids expensive lock overhead and thread safety issues by using CAS atomic instructions; in addition, inserting data points into the incremental buffer incrementally by using CAS atomic instructions reduces CPU cache invalidation.

[0095] The recursive model index in step (2) is established according to the following steps:

[0096] (A1) Obtain the dataset keys = {(key a ,pos a ), where a = 1, 2, ..., n}, where n represents the total number of data points in the dataset, i ∈ [1, n], key i pos represents the key of the i-th data point. i This indicates the position of the i-th data point in the dataset keys;

[0097] (A2) Set counter j = 1 and counter i = 2, and set the first data point (key1, pos1) in the dataset keys as the current starting point of leaf node j, and obtain point S according to the pre-set error threshold δ. a = (key1,pos1+δ) and point S b = (key1, pos1―δ);

[0098] Specifically, the error threshold δ in this step is any natural number. The smaller the value, the more leaf nodes are generated, which will affect the performance of the root node searching for leaf nodes. The larger the value, the larger the error of the linear regression model trained based on the data points managed by the leaf nodes. In this embodiment, the error threshold δ is preferably 16.

[0099] (A3) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8). Otherwise, generate point S based on the error threshold δ and the second data point (key2, pos2) in the dataset keys. c = (key2, pos2+δ) and point S d =(key2,pos2―δ), obtain point S a and point S d The generated line segment l1 and its slope slope1 are used to obtain point S. b and point S c Generate line segment l2 and its slope slope2, and obtain the intersection point S0(key0,pos0) of line segment l1 and line segment l2. Set the counter i = i + 1, and then proceed to step (A4).

[0100] (A4) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8). Otherwise, generate point S based on the error threshold δ and the i-th data point in the dataset keys. e =(key i ,pos i +δ) and point S f =(key i ,pos i ―δ), set counter i = i + 1, and calculate point S e The slope S between point S0 and point S0 high and point S f The slope S between point S0 and point S0 low And determine whether there is an S high ≤slope2 and slope1≤S low If yes, proceed to step (A5); otherwise, proceed to step (A8).

[0101] (A5) Determine if |pos exists. i ―slope1(key i If key0)pos0|>δ, then update. Then proceed to step (A6); otherwise, proceed to step (A6).

[0102] (A6) Determine if |pos exists. i ―slope2(key i If key0)pos0|>δ, then update. Then proceed to step (A7); otherwise, proceed to step (A7).

[0103] (A7) Set the counter i = i + 1 and return to step (A4);

[0104] (A8) Data point (key) i―1 ,pos i―1 All data points up to the current starting point are treated as data managed by the j-th leaf node, and it is determined whether the total number of all data points is greater than 1. If so, the slope of the linear regression model is set. Set the intercept of the linear regression model to intercept = pos0 - slope * key0, set the counter j = j + 1, and then proceed to step (A9). Otherwise, set the slope of the linear regression model to slope = 0 and the intercept to intercept = pos0. i―1 Set the counter j = j + 1, and then proceed to step (A9);

[0105] (A9) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A10); otherwise, remove the data point (key). i ,pos i Set the current starting point of leaf node j, and generate point S according to the error threshold. a = (key1,pos1+δ) and point S b = (key1, pos1―δ), then return to step (A3);

[0106] (A10) Select the starting point of each leaf node and the leaf node (i.e., ...<MinKey,Data Node> This is stored as metadata in the root node, thus creating the recursive model index.

[0107] For example, if the key values ​​of the data points managed by each leaf node are in the range [1,4), [4,65), [65,80), ..., [110,120), then the metadata stored in the root node is {(1,data_node1),(4,data_node2),(65,data_node3),...,(110,data_node4)}. Num )}, where Num represents the total number of leaf nodes included in the root node.

[0108] (3) Generate an incremental record based on the data point to be inserted obtained in step (2), insert the incremental record into the incremental linked list in the corresponding incremental buffer, and determine whether the number of incremental records stored in the incremental linked list reaches the preset threshold (the value range of the preset threshold is any positive integer, preferably 32). If it does, retrain the incremental linked list and send a notification of successful write operation to the user, and the process ends. Otherwise, send a notification of write operation to the user, and the process ends.

[0109] Specifically, incremental records are represented by triples of <write operation type, key, value>. For example, incremental record<insert 4,5> This indicates an insert operation, with key-value pairs 4 and 5; incremental record.<delete 4,5> This indicates a deletion operation, with key-value pairs 4 and 5.

[0110] The process of establishing the incremental buffer is as follows: an incremental buffer is allocated for each data point. The incremental buffer consists of four parts, including an incremental list, incremental nodes, a Bloom filter, and a structure adjuster. The incremental list and incremental nodes are initially empty.

[0111] Initializing all increment buffers incurs significant memory overhead when the key of a newly inserted increment record only covers a specific data range. To address this issue, a strategy of building increment buffers as needed is adopted. Specifically, when accessing an uninitialized increment buffer, the thread first initializes it and then uses a CAS atomic instruction to allocate the increment buffer to the corresponding data point.

[0112] like Figure 2 As shown, the process of inserting the incremental record into the incremental linked list in the incremental buffer in this step is as follows: First, the incremental record is pointed to the head of the incremental linked list through the linked list pointer. Then, the incremental record is inserted into the head of the incremental linked list through the CAS atomic instruction to complete the insertion operation (after multiple insertions, a series of incremental records are appended to the incremental buffer, forming a linked list structure called the incremental linked list).

[0113] In this step, retraining the incremental list involves merging the data points in the incremental list within the incremental buffer with the data points in the incremental nodes within the incremental buffer. For example... Figure 3 As shown, the process of merging the data points in the incremental linked list with the data points in the incremental node in this step includes the following sub-steps:

[0114] (3-1) Create two empty sets S insert and S delete S insert S is used to store the keys in incremental records where the write operation type is insert. delete Used to store keys in incremental records where the write operation type is delete;

[0115] (3-2) Scan the incremental linked list sequentially. Whenever an incremental record with key k and write operation type insertion operation is encountered (where k is a natural number), and k is not in set S, delete In the middle, k is added to set S. insert Meanwhile, whenever an incremental record with key value k and write operation type delete is scanned, and k is not in set S... insert In the middle, k is added to set S. delete middle;

[0116] (3-3) Set S insert and set S delete The merge is performed, and the merged results are sorted by key. Then, the sorted key-value pairs are merged with the key-value pairs in the incremental node to generate a new incremental node P containing all data points.

[0117] Specifically, the merging method is as follows;

[0118] Snew =S insert ∪S old -S delete

[0119] Where S old and S new These represent the sets of key-value pairs in the incremental node and the new incremental node, respectively.

[0120] (3-4) Replace the incremental node with the new incremental node P obtained in step (3-3) using the CAS atomic instruction;

[0121] (3-5) Determine whether the number of data points in the new incremental node obtained in step (3-4) is greater than or equal to the preset splitting threshold (in this embodiment, the value is equal to 512). If so, split the incremental node to obtain two new incremental nodes, and then proceed to step (3-7); otherwise, proceed to step (3-6).

[0122] The process of splitting the incremental node P is as follows: Figure 4 As shown:

[0123] (3-6) Determine whether the number of data points in the new incremental node obtained in step (3-4) is lower than the preset merging threshold (in this embodiment, the value is equal to 32). If so, merge the incremental node P with the sibling node located to the right of the incremental node in the recursive model index to obtain a new incremental node, and then proceed to step (3-12). Otherwise, the process ends.

[0124] The process of merging incremental node P and its sibling node is as follows: Figure 5 As shown:

[0125] (3-7) Divide the data points in the incremental node P obtained in step (3-4) equally, select the key b corresponding to the middle data point as the separator key, and create a new incremental buffer. The incremental buffer contains an incremental node Q. The incremental node P stores all the data points in the incremental node P whose keys are greater than or equal to the separator key b. Then proceed to step (3-8).

[0126] (3-8) Generate an incremental record Δsplit and set the incremental node Q as the sibling node to the right of the incremental record Δsplit in the recursive model index. Then insert the incremental record Δsplit into the incremental node P and determine whether other incremental records have been inserted into the incremental node P before the insertion of the incremental record Δsplit. If so, it means that the CAS atomic instruction failed to execute and then proceed to step (3-9). Otherwise, it means that the incremental record Δsplit was successfully inserted into the incremental node P and all data points in the incremental node P whose keys are greater than or equal to the separator key b in step (3-7) have become invalid. Proceed to step (3-10).

[0127] (3-9) Reclaim the incremental record Δsplit generated in step (3-8), divide the data points in the incremental node P obtained in step (3-4) into equal parts, select the key corresponding to the middle data point as the separator key, generate a new incremental record Δsplit, insert the incremental record Δsplit into the incremental node P, and then return to step (3-8).

[0128] (3-10) Insert an incremental record into the leaf node determined in step (2) (the purpose is to ensure that the incremental node Q can be directly searched), and then proceed to step (3-14).

[0129] (3-11) Obtain the sibling node R to the right of the incremental node P obtained in step (3-4) in the recursive model index, and generate an incremental record Δremove. Insert the incremental record Δremove into the incremental node R through the CAS atomic instruction.

[0130] The purpose of this step is to stop all further use of the incremental node R;

[0131] (3-12) Generate an incremental record Δmerge, which contains a physical pointer to the incremental node R in step (3-12), and insert the incremental record Δmerge into the incremental node P obtained in step (3-4) using the CAS atomic instruction, and then proceed to step (3-13).

[0132] The advantage of this step is that it achieves the logical merging of incremental nodes P and R in a non-blocking manner. The actual merging of incremental nodes can be completed by retraining the incremental linked list (the process is the same as steps (3-1)-(3-5) above, and will not be repeated here). Finally, a new incremental node is created, which contains all the data points of incremental nodes P and R, and incremental nodes P and R are recycled.

[0133] (3-13) Insert an incremental record into the leaf node obtained in step (2) (the purpose is to avoid searching for the incremental node R), and then proceed to step (3-14);

[0134] (3-14) Determine whether the number of data points in the incremental node P after the incremental record Δmerge is inserted in step (3-12) exceeds the preset threshold (the value of the preset threshold is any positive integer, preferably 256). If so, train the linear regression model based on the incremental node and then proceed to step (3-15). Otherwise, the process ends.

[0135] The advantage of steps (3-1) to (3-14) above is that the retraining operation in the incremental buffer is implemented in a non-blocking manner, reducing the overhead of the structure modification operation.

[0136] Specifically, the preset threshold value range is any positive integer, preferably 256;

[0137] (3-15) Obtain all data points in the incremental node P after the incremental record Δmerge was inserted in step (3-12), and use all data points as the dataset to train the linear regression model (the process is the same as steps (A1)-(A9) above, and will not be repeated here).

[0138] The advantage of this step is that it uses a linear regression model to accelerate the search process in incremental nodes, thereby improving search and insertion performance.

[0139] In concurrent scenarios, competition between threads can cause some CAS atomic instructions to fail, thus failing to guarantee that threads can make progress under all circumstances. This phenomenon is known as thread starvation. To alleviate thread starvation, the Help-along mechanism is introduced, which involves cooperation between threads to complete multi-stage incremental list retraining and incremental node retraining. Specifically, the Help-along mechanism requires that a thread must assist other threads in completing the incomplete incremental list retraining and incremental node retraining before executing its corresponding operation.

[0140] (4) Obtain the data point of the single-point query request and the incremental buffer where the single-point query request is located, and use the Bloom filter to determine whether the corresponding result of the single-point query request can be found in the incremental buffer. If yes, proceed to step (5); otherwise, return an empty result to the user and the process ends.

[0141] The process of obtaining the incremental buffer where the single-point query request is located is as follows: First, the root node in the recursive model index is searched using a binary search method to determine the leaf node corresponding to the single-point query request. Then, the predicted position of the data point of the single-point query request among all data points managed by the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using a binary search method, the true position of the data point of the single-point query request in the leaf node is determined. The incremental buffer associated with this true position is the incremental buffer where the single-point query request is located.

[0142] The advantage of this step is that the existence of data points can be quickly determined using a Bloom filter, reducing unnecessary memory accesses.

[0143] The incremental buffer consists of four parts: an incremental list and incremental nodes for storing data points, a Bloom filter for quickly determining whether the data point requested by the query exists, and a structure adjuster for guiding the structure adjustment operations in the incremental buffer, namely, incremental list retraining, incremental node splitting and merging.

[0144] (5) Determine whether the result corresponding to the single-point query request can be found in the incremental linked list in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, proceed to step (6).

[0145] Specifically, the process of querying the result corresponding to the single-point query request in the incremental linked list in this step is as follows: sequentially scan the incremental linked list and determine whether the key of the currently scanned incremental record is the same as the key of the data point corresponding to the single-point query request.

[0146] (6) Determine whether the result corresponding to the single-point query request can be found in the incremental node in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, return an empty result to the user and the process ends.

[0147] Specifically, the process of querying the result corresponding to the single-point query request in the incremental node is as follows: If the number of incremental data records in the incremental node is less than a preset threshold (preferably 256), the incremental node is searched using a binary search method to determine the true position of the data point of the single-point query request in the incremental node; otherwise, firstly, the predicted position of the key of the data point of the single-point query request in the incremental node is calculated using the linear regression model corresponding to the incremental node. Finally, based on the predicted position and the error range of the linear regression model, the true position of the data point corresponding to the single-point query request in the incremental node is determined using a binary search method, and the data point at the true position is obtained as the query result.

[0148] (7) Obtain the starting value and ending value corresponding to the range query request and the leaf node where the starting value of the range query request is located. Determine the starting position of the starting value among all data points managed by the leaf node. Starting from the starting position, traverse all data points managed by the leaf node whose keys are between the starting value and the ending value. Store all data points in the result set R1 and perform further search in the incremental linked list and incremental node in the incremental buffer associated with the starting position to obtain all data points whose keys are between the starting value and the ending value. Then proceed to step (8).

[0149] Specifically, the process of determining the starting position of the initial value among all data points in this step is as follows: First, the root node in the recursive model index is searched using a binary search method to determine the leaf node corresponding to the starting value of the range query request. Then, the predicted position of the starting value of the range query request in the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using a binary search method to determine the true position of the starting value of the range query request in the leaf node, the true position is taken as the starting position, and the incremental buffer associated with the starting position is taken as the further search object.

[0150] The incremental buffer consists of four parts: an incremental list and incremental nodes for storing data points, a Bloom filter for quickly determining whether the data point requested by the query exists, and a structure adjuster for guiding the structure adjustment operations in the incremental buffer, namely, incremental list retraining, incremental node splitting and merging.

[0151] (8) Obtain all incremental linked lists in the incremental buffer obtained in step (7), traverse all incremental linked lists, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R2. Merge the result set R2 with R1 to obtain a new result set R3.

[0152] During the merging process in this step, if the same data point (i.e., the corresponding key is the same in result sets R1 and R2) is used, then only the data point in result set R2 will be stored in R3.

[0153] (9) Obtain all incremental nodes in the incremental buffer obtained in step (7), traverse all incremental nodes, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R4. Merge the result set R4 with R3 to obtain a new result set R5.

[0154] During the merging process in this step, if the same data point (i.e., the corresponding key is the same in result sets R3 and R4) is used, then only the data point in result set R4 will be saved in R5.

[0155] Experimental results

[0156] The experimental environment for this invention was as follows: two 16-core Intel(R) Xeon(R) Gold 6142 CPUs at 2.60GHz, 384GB of DDR4 memory, and a 2TB hard drive. The system was implemented in C++ on a 64-bit Ubuntu 18.04.1LTS operating system. Specific parameter settings were as follows: the threshold for the number of records stored in the incremental linked list was set to 32, the maximum threshold for the number of records stored in the incremental node was set to 512, and the minimum threshold for the number of records was set to 32.

[0157] To demonstrate the high read and write performance of this invention, read and write load tests were conducted on four datasets: Normal, Lognormal, Facebook, and OSM. The read throughput and write throughput of each index structure on different datasets were recorded. Figure 4 The read load test results for various index structures under different threads are presented. Figure 5 Write load test results for various index structures under different thread conditions are presented (where LFDex represents the present invention). These two figures show that the present invention's solution exhibits the best read and write performance in high-concurrency scenarios: Under 32 threads, compared to Masstree, the present invention improves read performance by 1.3 times and write performance by 3.4 times; compared to BwTree, the present invention improves read performance by 1.6 times and write performance by 3.6 times; compared to XIndex, the present invention improves read performance by 1.4 times and write performance by 1.1 times; and compared to FINEdex, the present invention improves read performance by 1.4 times and write performance by 0.9 times. With 64 threads, compared to Masstree, the read performance of this invention is improved by 1.5 times and the write performance by 7.9 times; compared to BwTree, the read performance is improved by 2.9 times and the write performance by 5.7 times; compared to XIndex, the read performance is improved by 1.6 times and the write performance by 3.6 times; and compared to FINEdex, the read performance is improved by 1.2 times and the write performance by 6.3 times.

[0158] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A learning indexing method supporting high-concurrency scenarios, characterized in that, Includes the following steps: (1) Receive a request from the user and determine whether the request is a single-point query request, a range query request, or a write request based on the interface through which the user sent the request. If it is a single-point query request, proceed to step (4); if it is a range query request, proceed to step (7); if it is a write request, proceed to step (2). (2) Obtain the data point to be inserted according to the write request, and determine the leaf node corresponding to the data point in the pre-established recursive model index; (3) Generate an incremental record based on the data point to be inserted obtained in step (2), insert the incremental record into the incremental linked list in the corresponding incremental buffer, and determine whether the number of incremental records stored in the incremental linked list has reached the preset threshold. If so, retrain the incremental linked list and send a notification of successful write operation to the user. The process ends. Otherwise, send a notification of write operation to the user. The process ends. In step (3), the process of inserting the incremental record into the incremental linked list in the incremental buffer is as follows: First, the incremental record is pointed to the head of the incremental linked list through the linked list pointer. Then, the incremental record is inserted into the head of the incremental linked list through the CAS atomic instruction to complete the insertion operation. Step (3) involves retraining the incremental list by merging the data points in the incremental list in the incremental buffer with the data points in the incremental nodes of the incremental buffer. This process includes the following sub-steps: (3-1) Create two empty sets and ,in Used to store keys in incremental records where the write operation type is insert. Used to store keys in incremental records where the write operation type is delete; (3-2) Scan the incremental linked list sequentially. Whenever an incremental record with key k and write operation type insertion is encountered, and k is not in the set... In the middle, k is added to the set. Meanwhile, whenever an incremental record with key value k and write operation type delete is scanned, and k is not in the set... In the middle, k is added to the set. In the case of k, k is a natural number; (3-3) Set and set The merge is performed, and the merged results are sorted by key. Then, the sorted key-value pairs are merged with the key-value pairs in the incremental node to generate a new incremental node P containing all data points. (3-4) Replace the incremental node with the new incremental node P obtained in step (3-3) using the CAS atomic instruction; (4) Obtain the data point of the single-point query request and the incremental buffer where the single-point query request is located, and use the Bloom filter to determine whether the corresponding result of the single-point query request can be found in the incremental buffer. If yes, proceed to step (5); otherwise, return an empty result to the user and the process ends. (5) Determine whether the result corresponding to the single-point query request can be found in the incremental linked list in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, proceed to step (6). (6) Determine whether the result corresponding to the single-point query request can be found in the incremental node in the incremental buffer obtained in step (4). If yes, return the result to the user and the process ends; otherwise, return an empty result to the user and the process ends. (7) Obtain the starting and ending values ​​corresponding to the range query request and the leaf node where the starting value of the range query request is located. Determine the starting position of the starting value among all data points managed by the leaf node. Starting from the starting position, traverse all data points managed by the leaf node whose keys are between the starting and ending values. Store all data points in the result set R1 and perform further search in the incremental linked list and incremental node in the incremental buffer associated with the starting position to obtain all data points whose keys are between the starting and ending values. Then proceed to step (8). (8) Obtain all incremental linked lists in the incremental buffer obtained in step (7), traverse all incremental linked lists, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R2. Merge the result set R2 with R1 to obtain a new result set R3. (9) Obtain all incremental nodes in the incremental buffer obtained in step (7), traverse all incremental nodes, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R4. Merge the result set R4 with R3 to obtain a new result set R5.

2. The learning indexing method supporting high-concurrency scenarios according to claim 1, characterized in that, In step (1), the specific type of the request is determined by checking the interface through which the user sends the request. If it is a single-point query request, the user sends the request through the single-point query interface. If it is a range query request, the user sends the request through the range query interface. If it is a write request, the user sends the request through the write program interface. Step (2) is to query the leaf node corresponding to the data point in the recursive model index. Specifically, the process is as follows: First, the binary search method is used to determine the real position of the key of the data point in the recursive model index, and the leaf node at the real position is obtained as the further search object.

3. The learning indexing method supporting high-concurrency scenarios according to claim 1 or 2, characterized in that, The recursive model index in step (2) is established according to the following steps: (A1) Obtain the dataset Where n represents the total number of data points in the dataset. , This represents the key of the i-th data point. This indicates the position of the i-th data point in the dataset keys; (A2) Set counter j=1, counter i=2, and set the first data point in the dataset keys. Set as the current starting point of leaf node j, and based on a pre-set error threshold. Acquisition Points and points ;wherein the error threshold It is any natural number. The smaller its value, the more leaf nodes are generated, which will affect the performance of the root node searching for leaf nodes. The larger its value, the larger the error of the linear regression model trained based on the data points managed by the leaf nodes. (A3) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8); otherwise, determine the error threshold. and the second data point in the dataset keys Generating points and points Acquisition Points and points The generated line segments and its slope Acquisition Points and points The generated line segments and its slope and obtain line segments With line segment intersection Set the counter i = i + 1, and then proceed to step (A4). (A4) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A8); otherwise, determine the error threshold. Generate a point from the i-th data point in the dataset keys. and points Set the counter i = i + 1, and calculate the point. With point The slope between and points With point The slope between And determine whether there is as well as If yes, proceed to step (A5); otherwise, proceed to step (A8). (A5) Determine if there is If so, then update. If yes, proceed to step (A6); otherwise, proceed to step (A6). (A6) Determine if there is If so, then update. If yes, proceed to step (A7); otherwise, proceed to step (A7). (A7) Set counter i = i + 1 and return to step (A4); (A8) Data points All data points up to the current starting point are treated as data managed by the j-th leaf node, and it is determined whether the total number of all data points is greater than 1. If so, the slope of the linear regression model is set. Set the intercept of the linear regression model Set the counter j = j + 1, then proceed to step (A9); otherwise, set the slope of the linear regression model. ,intercept Set the counter j = j + 1, and then proceed to step (A9). (A9) Determine if i is greater than the total number of data points n in the dataset. If yes, proceed to step (A10); otherwise, remove the data points. Starting point, and generating points based on the error threshold. and points Then return to step (A3); (A10) Select the starting point of each leaf node and store the leaf node as metadata in the root node to obtain the recursive model index.

4. The learning indexing method supporting high-concurrency scenarios according to claim 3, characterized in that, Incremental records are represented by triples of <write operation type, key, value>. The process of establishing the incremental buffer is as follows: an incremental buffer is allocated for each data point. The incremental buffer consists of four parts, including an incremental list, incremental nodes, a Bloom filter, and a structure adjuster. The incremental list and incremental nodes are initially empty.

5. The learning indexing method supporting high-concurrency scenarios according to claim 4, characterized in that, The retraining process of the incremental linked list in step (3) further includes the following sub-steps: (3-5) Determine whether the number of data points in the new incremental node obtained in step (3-4) is greater than or equal to the preset splitting threshold. If so, split the incremental node to obtain two new incremental nodes, and then proceed to step (3-7). Otherwise, proceed to step (3-6). (3-6) Determine whether the number of data points in the new incremental node obtained in step (3-4) is lower than the preset merging threshold. If so, merge the incremental node P with the sibling node located to the right of the incremental node in the recursive model index to obtain a new incremental node, and then proceed to step (3-12). Otherwise, the process ends. (3-7) Divide the data points in the incremental node P obtained in step (3-4) equally, select the key b corresponding to the middle data point as the separator key, and create a new incremental buffer. The incremental buffer contains an incremental node Q. The incremental node P stores all the data points in the incremental node P whose keys are greater than or equal to the separator key b. Then proceed to step (3-8). (3-8) Generate an incremental record And set the incremental node Q as the incremental record in the recursive model index. The sibling node on the right, then the incremental record. Insert the incremental record into node P, and determine if the incremental record has been inserted. If any other incremental records have been inserted into the incremental node P before, it indicates that the CAS atomic instruction failed, and then proceed to step (3-9); otherwise, it indicates that the incremental record... Successfully inserted into incremental node P, and all data points in incremental node P whose keys are greater than or equal to the separator key b in step (3-7) have become invalid, proceed to step (3-10). (3-9) Recycling the incremental records generated in step (3-8) The data points in the incremental node P obtained in step (3-4) are divided equally, and the key corresponding to the middle data point is selected as the separator key to generate a new incremental record. And record the increment. Insert into the incremental node P, and then return to step (3-8); (3-10) Insert an incremental record into the leaf node determined in step (2), and then proceed to step (3-14). (3-11) Obtain the sibling node R to the right of the incremental node P obtained in step (3-4) in the recursive model index, and generate an incremental record. The increment is recorded using CAS atomic instructions. Insert into the incremental node R; (3-12) Generate an incremental record The increment record contains a physical pointer to the increment node R in step (3-12), and the increment record is executed via a CAS atomic instruction. Insert it into the incremental node P obtained in step (3-4), and then proceed to step (3-13). (3-13) Insert an incremental record into the leaf node obtained in step (2), and then proceed to step (3-14). (3-14) Determine if an incremental record was inserted in step (3-12). If the number of data points in the incremental node P exceeds the preset threshold, then train a linear regression model based on the incremental node and proceed to step (3-15); otherwise, the process ends. (3-15) The incremental record was inserted in step (3-12). The incremental node P contains all data points, and all data points are used as the dataset to train the linear regression model.

6. The learning indexing method supporting high-concurrency scenarios according to claim 5, characterized in that, The merging method in step (3-3) is as follows; ； in and These represent the sets of key-value pairs in the incremental node and the new incremental node, respectively.

7. The learning indexing method supporting high-concurrency scenarios according to claim 1, characterized in that, The process of obtaining the incremental buffer where the single query request is located in step (4) is as follows: First, the root node in the recursive model index is searched using the binary search method to determine the leaf node corresponding to the single query request. Then, the predicted position of the data point of the single query request in all the data points managed by the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using the binary search method, the true position of the data point of the single query request in the leaf node is determined. The incremental buffer associated with the true position is the incremental buffer where the single query request is located. The process of querying the result corresponding to the single-point query request in the incremental node in step (6) is as follows: If the number of incremental data records in the incremental node is less than the preset threshold, the binary search method is used to search the incremental node to determine the real position of the data point of the single-point query request in the incremental node; otherwise, firstly, the linear regression model corresponding to the incremental node is used to calculate the predicted position of the key of the data point of the single-point query request in the incremental node. Finally, based on the predicted position and the error range of the linear regression model, the binary search method is used to determine the real position of the data point corresponding to the single-point query request in the incremental node, and the data point at the real position is obtained as the query result. In step (7), the process of determining the starting position of the starting value among all data points is as follows: First, the root node in the recursive model index is searched using the binary search method to determine the leaf node corresponding to the starting value of the range query request. Then, the predicted position of the starting value of the range query request in the leaf node is calculated using the linear regression model corresponding to the leaf node. Finally, based on the predicted position and the error range of the linear regression model, and using the binary search method to determine the true position of the starting value of the range query request in the leaf node, the true position is taken as the starting position, and the incremental buffer associated with the starting position is taken as the further search object.

8. A learning indexing system supporting high-concurrency scenarios, implemented based on the learning indexing method for supporting high-concurrency scenarios as described in claim 1, characterized in that, The learning index system includes: The first module is used to receive requests from users and determine whether the request is a single-point query request, a range query request, or a write request based on the interface through which the user sent the request. If it is a single-point query request, it proceeds to the fourth module; if it is a range query request, it proceeds to the seventh module; and if it is a write request, it proceeds to the second module. The second module is used to obtain the data point to be inserted according to the write request, and determine the leaf node corresponding to the data point in the pre-established recursive model index. The third module is used to generate an incremental record based on the data point to be inserted obtained from the second module, insert the incremental record into the incremental linked list in the corresponding incremental buffer, and determine whether the number of incremental records stored in the incremental linked list has reached a preset threshold. If so, the incremental linked list is retrained and a notification of successful write operation is sent to the user, and the process ends. Otherwise, a notification of successful write operation is sent to the user, and the process ends. The fourth module is used to obtain the data point of the single-point query request and the incremental buffer where the single-point query request is located, and to use a Bloom filter to determine whether the corresponding result of the single-point query request can be found in the incremental buffer. If so, proceed to the fifth module; otherwise, return an empty result to the user and the process ends. The fifth module is used to determine whether the result corresponding to the single-point query request can be found in the incremental linked list in the incremental buffer obtained in the fourth module. If so, the result is returned to the user and the process ends; otherwise, the process proceeds to the sixth module. The sixth module is used to determine whether the result corresponding to the single-point query request can be found in the incremental node in the incremental buffer obtained by the fourth module. If so, the result is returned to the user and the process ends; otherwise, an empty result is returned to the user and the process ends. The seventh module is used to obtain the start and end values ​​corresponding to the range query request and the leaf node where the start value of the range query request is located. It determines the corresponding starting position of the start value among all data points managed by the leaf node. Starting from the starting position, it traverses all data points managed by the leaf node whose keys are between the start and end values, and stores all data points in the result set R1. It then performs further searches in the incremental linked list and incremental nodes in the incremental buffer associated with the starting position to obtain all data points whose keys are between the start and end values, and then proceeds to the eighth module. The eighth module is used to obtain all incremental linked lists in the incremental buffer obtained by the seventh module, traverse all incremental linked lists, extract all data points whose keys are between the start value and the end value, and store all data points in the result set R2. The result set R2 is merged with R1 to obtain a new result set R3. The ninth module is used to obtain all incremental nodes in the incremental buffer obtained by the seventh module, traverse all incremental nodes, extract all data points whose keys are between the start value and the end value, and save all data points in the result set R4. The result set R4 is merged with R3 to obtain a new result set R5.

Images