Index processing method and device

Abstract

The invention discloses an index processing method and device, and relates to the technical field of data processing. A specific embodiment of the method comprises the following steps: receiving a key to be processed, determining a projection position of the key in a root node by using a model index, determining a pointer corresponding to the projection position from an array containing a pointer pointing to a child node, and determining the child node; the internal nodes repeatedly operate until the data nodes are determined; under the condition that the data node is inserted, judging whether the data node meets a preset expansion splitting condition or not, if so, calling a preset node expansion splitting mechanism, and executing an expansion splitting operation on the data node; and after the expansion splitting is finished or is not consistent, inserting the to-be-inserted data into the projection position in the gap array, or reading a pointer corresponding to the projection position, and inserting the to-be-inserted data into a memory storage position pointed by the pointer. According to the implementation mode, rapid and accurate data insertion can be realized, and the performance bottleneck caused by data node overload is avoided through a node expansion splitting mechanism.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an index processing method and device. Background Art

[0002] In the face of the storage, retrieval and modification needs of massive data, efficient database indexing (Index) becomes the key. Indexes significantly improve data retrieval efficiency, reduce I / O costs, and ensure data uniqueness and query stability by using data structures such as B-trees or B+ trees. In particular, B+ trees, because their non-leaf nodes do not store keyword record pointers, can use space more efficiently and further improve query performance. However, indexes also have problems such as taking up a lot of disk space and high creation and maintenance costs. Although some studies have proposed using hierarchical machine learning models to replace standard indexes, this solution is currently only applicable to read-only data and does not support data update and insertion operations. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an index processing method and device, which can at least solve the problem in the prior art that only data can be read but data update and insertion operations are not supported.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, an index processing method is provided, comprising:

[0005] Receive the key to be processed, determine the projection position of the key in the root node using the model index, determine the pointer corresponding to the projection position from the array of pointers to child nodes in the root node, and then determine the child node according to the pointer;

[0006] In response to the type of the child node being an internal node, repeating the above operations of determining the projection position, the pointer, and the child node until the type of the child node determined according to the pointer is a data node;

[0007] In response to the type of the child node being a data node, when the processing type is insert, determining whether the data node meets a preset expansion and splitting condition, and when the determination result is that the condition is met, calling a preset node expansion and splitting strategy to perform an expansion and splitting operation on the data node;

[0008] After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or the pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

[0009] Optionally, the method further comprises:

[0010] When the processing type is query, the model index is used to determine the projection position of the key, and the data record corresponding to the projection position is read from the gap array and returned, or the pointer corresponding to the projection position is read from the gap array, and then the data record is read from the corresponding storage location in the memory according to the pointer and returned.

[0011] Optionally, the determining whether the data node meets a preset expansion and splitting condition includes:

[0012] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0013] In response to the total number of keys being greater than a preset threshold, it is determined that the data node meets a preset expansion split condition.

[0014] Optionally, calling a preset node expansion and splitting strategy to perform an expansion and splitting operation on a data node includes:

[0015] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0016] Divide the total number of keys by a preset value to obtain a new number of keys, split the new number of keys into two parts in order, and distribute them to two new data nodes; and

[0017] Update the parameters in the model index of the data node based on the preset value to obtain a new model index, and then determine the new mapping position of each key according to the new model index;

[0018] Store the pointer or data record of each key in the data node in the new data node according to the new mapping position.

[0019] Optionally, the method further comprises:

[0020] Using a preset loss function, calculate the current loss value of the current tree structure; wherein the current tree structure includes internal nodes and data nodes, and the internal nodes include the root node;

[0021] Use the preset tuner to adjust the parameters in the model index, use the model index with updated parameters to process all internal nodes and data nodes in the tree structure, and obtain a new tree structure;

[0022] Using a preset loss function, a new loss value of the new tree structure is calculated, and in response to the new loss value being less than the current loss value, the current tree structure is replaced with the new tree structure.

[0023] Optionally, the parameters for calculating the loss value include one or more of the following:

[0024] Get the historical processing records and get the depth of each historical processing record traversed to the data node; the depth is the level from the root node to the gap array;

[0025] Get the memory size occupied by all internal nodes and data nodes;

[0026] For the historical processing records whose processing type is insert, the number of shifts is obtained, and then the average number of shifts is obtained.

[0027] To achieve the above object, according to another aspect of an embodiment of the present invention, an index processing device is provided, comprising:

[0028] A receiving module, used for receiving keys to be processed;

[0029] A node processing module, used to determine the projection position of the key in the root node using the model index, determine the pointer corresponding to the projection position from the array of the root node containing pointers to the child nodes, and then determine the child node according to the pointer;

[0030] In response to the type of the child node being an internal node, repeating the above operations of determining the projection position, the pointer, and the child node until the type of the child node determined according to the pointer is a data node;

[0031] The insert module is used for, in response to the type of the child node being a data node, judging whether the data node meets the preset expansion and splitting condition when the processing type is insertion, and if the judgment result is that the condition is met, calling the preset node expansion and splitting strategy to perform the expansion and splitting operation on the data node;

[0032] After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or the pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

[0033] Optionally, the device further comprises a query module, configured to:

[0034] When the processing type is query, the model index is used to determine the projection position of the key, and the data record corresponding to the projection position is read from the gap array and returned, or the pointer corresponding to the projection position is read from the gap array, and then the data record is read from the corresponding storage location in the memory according to the pointer and returned.

[0035] Optionally, the insertion module is used to:

[0036] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0037] In response to the total number of keys being greater than a preset threshold, it is determined that the data node meets a preset expansion split condition.

[0038] Optionally, the insertion module is used to:

[0039] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0040] Divide the total number of keys by a preset value to obtain a new number of keys, split the new number of keys into two parts in order, and distribute them to two new data nodes; and

[0041] Update the parameters in the model index of the data node based on the preset value to obtain a new model index, and then determine the new mapping position of each key according to the new model index;

[0042] Store the pointer or data record of each key in the data node in the new data node according to the new mapping position.

[0043] Optionally, the device further comprises a model updating module, configured to:

[0044] Using a preset loss function, calculate the current loss value of the current tree structure; wherein the current tree structure includes internal nodes and data nodes, and the internal nodes include the root node;

[0045] Use the preset tuner to adjust the parameters in the model index, use the model index with updated parameters to process all internal nodes and data nodes in the tree structure, and obtain a new tree structure;

[0046] Using a preset loss function, a new loss value of the new tree structure is calculated, and in response to the new loss value being less than the current loss value, the current tree structure is replaced with the new tree structure.

[0047] Optionally, the parameters for calculating the loss value include one or more of the following:

[0048] Get the historical processing records and get the depth of each historical processing record traversed to the data node; the depth is the level from the root node to the gap array;

[0049] Get the memory size occupied by all internal nodes and data nodes;

[0050] For the historical processing records whose processing type is insert, the number of shifts is obtained, and then the average number of shifts is obtained.

[0051] To achieve the above objective, according to another aspect of an embodiment of the present invention, an index processing electronic device is provided.

[0052] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned index processing methods.

[0053] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above index processing methods is implemented.

[0054] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product according to an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the index processing method provided by the embodiment of the present invention is implemented.

[0055] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: the characteristics of the cumulative distribution function can be efficiently utilized, combined with different types of model indexes, to achieve fast and accurate data insertion. For data insertion, the load balancing of data nodes is ensured through the node expansion and splitting mechanism, avoiding the performance bottleneck caused by the overload of a single data node. In addition, in the case of data node expansion and splitting, the high efficiency of data insertion and query is guaranteed by dynamically adjusting the parameters of the model index, which significantly improves the overall performance of the system.

[0056] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0058] Figure 1 It is an existing schematic diagram of using machine learning models to replace standard indexes;

[0059] Figure 2 This is a schematic diagram of the index query of the model of this solution;

[0060] Figure 3 is a schematic diagram of the main flow of an index processing method according to an embodiment of the present invention;

[0061] Figure 4 It is a schematic diagram of the internal node index;

[0062] Figure 5 A schematic flow chart of an optional index processing method according to an embodiment of the present invention;

[0063] Figure 6 is a flowchart of another optional index processing method according to an embodiment of the present invention;

[0064] Figure 7 is a schematic diagram of main modules of an index processing device according to an embodiment of the present invention;

[0065] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0066] Fig. 9 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0068] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0069] In the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The acquisition, transmission, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations, and are used for legal and reasonable purposes, and are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities.

[0070] For user information, necessary measures should be taken to prevent illegal access to such personal information data, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data. When applicable, including in certain relevant applications, protect user privacy by de-identifying data, such as by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the specific address level), controlling how data is stored, and / or other methods of de-identification.

[0071] In the face of the need to store, retrieve, and modify massive amounts of data, efficient database indexing becomes the key. For example, when processing the issuance and collection of a large number of consumer coupons, each preferential activity will generate massive amounts of data, which requires efficient storage, retrieval, and modification operations. Due to the large amount of data, it will be impossible to complete the query in a short period of time, which greatly reduces the user experience. For this reason, database indexing becomes a key tool. An index is a data structure that helps the database to efficiently obtain data. By using B-tree or B+ tree data structures, it significantly improves data retrieval efficiency, reduces the I / O cost of the database, and ensures the uniqueness of each row of data in the database table and the stability of the query.

[0072] In particular, the B+ tree is a balanced multi-way search tree. The keywords of all nodes follow the principle of small on the left and large on the right. This structure makes the query speed more stable. Compared with other data structures, such as binary trees, red-black trees, hash tables, etc., the non-leaf nodes of the B+ tree do not store keyword record pointers, which can make better use of space and further improve query performance. Traditional databases such as MySQL's MyISAM and InnoDB engines all use B+ trees as index storage structures.

[0073] However, indexes also have their disadvantages, for example, they take up a lot of disk space; creating and maintaining indexes takes a lot of time; indexes need to be dynamically maintained when adding, deleting, and modifying data in a table, and as the amount of data increases, the maintenance cost will also increase significantly. Although some studies have proposed using hierarchical machine learning models to replace standard indexes, this solution is currently only applicable to situations where read-only data can be processed and does not support data update operations.

[0074] Logically, an index can be considered a "model" for predicting the location of data: that is, given a key, predict the location of the key in the sorted array of leaf nodes. Therefore, machine learning techniques can be used to generate model indexes optimized for specific data distributions by learning the cumulative distribution function (CDF) of the input data. The cumulative distribution function of a key is a function used to describe the probability that a random variable is less than or equal to a certain value. In data analysis and statistics, CDF can help understand the distribution characteristics of data. Suppose there is a random variable (X) whose possible values ​​are certain key values ​​(for example, the values ​​in a data set), then the cumulative distribution function (F(x)) is defined as: [F(x)=P(X \leq x)] where (P(X \leq x)) represents the probability that the random variable (X) takes a value less than or equal to (x).

[0075] Some important properties of CDF include: 1. Monotonicity: CDF is a non-decreasing function. 2. Boundary conditions: When (x) approaches negative infinity, the value of (F(x)) approaches 0; when (x) approaches positive infinity, the value of (F(x)) approaches 1. 3. Continuity: For continuous random variables, CDF is a continuous function. Through CDF, we can obtain a lot of information about random variables, such as the shape of its distribution, the center position, and the degree of dispersion. In practical applications, CDF is often used in probability calculation, threshold analysis and other scenarios.

[0076] Based on this idea, it is now proposed to use hierarchical machine learning models to replace standard database indexes. Figure 1 The figure shows a schematic diagram of the existing use of machine learning models to replace standard indexes. Given a key, the intermediate node model is used to predict the sub-model to be used next, for example, the root model determines to use Model1.2, and Model 1.2 determines to use Model 2.3; the leaf node model is used to predict the position of the key in the array, for example, Model 2.3 determines the position of the key. This model is trained with data, and as long as a suitable distribution model is used, the position of the key can be predicted, thereby significantly improving the performance of data retrieval. The model index currently used belongs to the linear regression model: [y=ax + b] where (x) is the key and (y) is the predicted position. Since only two parameters (a) and (b) need to be stored, the storage overhead is very low and the execution speed is fast.

[0077] However, this solution can only handle read-only data and does not support data update operations. For example, given a key to be inserted or deleted, first use the model to find the insertion position of the key, then create a new array with a length of 1 plus the length of the original array, then copy the data from the old array to the new array, insert the key into the new array, and finally update the model to reflect the change in data distribution. This strategy is too complicated when dealing with database tables that are frequently inserted and deleted, because the model needs to be updated every time a piece of data is inserted or deleted.

[0078] This solution can be considered as a further optimization based on the existing technology. Because the model index of each level can be different types of models, such as nonlinear function model, linear function model, cumulative distribution function model, this solution first divides the nodes into two categories: data node and internal node.

[0079] Data Node: Like the B+ tree, the leaf nodes store data records, so they are called data nodes (such as Figure 2The data node stores a model index, such as a linear regression model, which maps a key to a position and an array with gaps. This array can store a specific record, such as a specific content, or a pointer to a variable-size record. The pointer points to the specific content in the memory, and the required content can be obtained through the pointer.

[0080] Figure 2 The horizontal line in the circle represents the array. The amount of data in the array is represented by black. Black means there is data, and white means there is no data stored in it. Among them, the array is a data structure used in encoding. The data with gaps means that the data is not full yet and you can continue to store data in it. Arrays can be understood as some containers that can store data of a certain size (the data here can be keys, pointers, functions, parameters, etc.).

[0081] Internal Node: Each node with a model index is called an internal node (such as Figure 2 The internal node stores a cumulative distribution function model (such as a linear regression model) and an array containing pointers to child nodes. Unlike the B+ tree, the internal node uses a flexible model to calculate the position of the next child pointer in the pointer array.

[0082] See also Figure 3 , which shows a main flow chart of an index processing method provided by an embodiment of the present invention, comprising the following steps:

[0083] S301: receiving a key to be processed, determining a projection position of the key in a root node using a model index, determining a pointer corresponding to the projection position from an array of pointers to child nodes in the root node, and then determining the child node according to the pointer;

[0084] S302: In response to the type of the child node being an internal node, repeat the above operations of determining the projection position, pointer, and child node until the type of the child node determined according to the pointer is a data node;

[0085] S303: In response to the type of the child node being a data node, when the processing type is insert, determining whether the data node meets a preset expansion and splitting condition;

[0086] S304: If the result of the judgment is in compliance, calling a preset node expansion and splitting strategy to perform an expansion and splitting operation on the data node;

[0087] S305: After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or a pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

[0088] Division and search of internal nodes and data nodes, Figure 4 The example shown here is a simple example. The actual operation is more complicated.

[0089] Assume that the cumulative distribution function of the key in the internal node A (i.e., the parent node A) is as follows Figure 4 (right) shows that the key range is [0, 1). Model indexing assigns different key ranges to data nodes and internal nodes based on the characteristics of the cumulative distribution function. Specifically: The key ranges [0, 1 / 4) and [1 / 2, 1) are assigned to data nodes: Because the cumulative distribution function in these ranges is linear, a simple linear model can be used directly for prediction. The key range [1 / 4, 1 / 2) is assigned to another internal node: Because the cumulative distribution function in this range is nonlinear, a more complex model is required for accurate prediction.

[0090] See also Figure 4 As shown, the cumulative distribution function of the key of internal node A is evenly divided into four ranges: [0, 1 / 4), [1 / 4, 1 / 2), [1 / 2, 3 / 4), [3 / 4, 1). The specific allocation is as follows: The first key range [0, 1 / 4): assigned to data node B, because the cumulative distribution function in this range is linear. The second key range [1 / 4, 1 / 2): assigned to internal node C, because the cumulative distribution function in this range is nonlinear. The third key range [1 / 2, 3 / 4) and the fourth key range [3 / 4, 1): these two ranges are continuous and linear, so the entire range [1 / 2, 1) is assigned to data node D.

[0091] During the search process, if the key in the parent node A is mapped to a data node, the search is complete and the specific location of the key can be directly found in the data node, or an insert or update operation can be performed. If the mapping results in an internal node, the cumulative distribution function of the internal node needs to be used to continue the mapping search until the data node is finally reached.

[0092] In the above implementation, for steps S301 to S302, a key to be processed is received. Assuming a data query scenario, a key is given. First, the model index is used to iteratively calculate from the root node. Taking the model index as a linear regression model as an example, the key is input into the linear regression model to determine the key range in which the key falls, that is, to determine the projection position of the key, such as 1 / 2 in Figure 4 The projection position in the middle right. According to the key range, the pointer corresponding to the key range is determined from the array of root node pointers to child nodes. For example, according to the above description, the array of parent node A includes three pointers. The first pointer corresponds to the key range [0,1 / 4) and points to data node B. The second pointer corresponds to the key range [1 / 4, 1 / 2) and points to the internal node C of the next level. The third pointer corresponds to the key range [1 / 2, 1) and points to data node D.

[0093] Assume that the key to be processed is 1 / 2, and the key range it falls into is determined to be [1 / 2, 1). The third pointer corresponding to [1 / 2,1) is determined from the array, pointing to data node D, and the model index in data node D is used to predict the position of key 1 / 2. However, if the key to be processed is 1 / 4, the key range it falls into is [1 / 4, 1 / 2). The second pointer corresponding to [1 / 4, 1 / 2) is determined from the array, pointing to data node C, that is, the pointer reaches the internal node C of the next layer. Repeat the above operations of determining the projection position, pointer, and child node at the internal node C until the final data node is reached, and then use the model index in the data node to predict the position of key 1 / 4. In this way, the characteristics of the cumulative distribution function can be efficiently utilized, combined with different types of model indexes, to achieve fast and accurate data retrieval.

[0094] For steps S303 to S305, for data query, the projection position of the key is determined by using the model index, such as y=ax+b, and then the data record corresponding to the projection position is read from the array of data nodes and returned. Alternatively, the pointer corresponding to the projection position is read from the array of data nodes to read and return the data record from the corresponding storage position in the memory based on the pointer. For data insertion, the logic of reaching the correct data node is the same as the logic of the above search algorithm. If there is still a gap at the insertion position, the key is inserted into the gap to complete the task.

[0095] However, the above is for the case where the array is not full, that is, the sum of the number of keys stored in the data node and the number of keys to be inserted this time is less than or equal to the preset threshold. For example, if the preset threshold is set to 1000, 900 keys are stored in the current data node, and 20 new keys need to be inserted this time, the total number is 920<1000, so it does not meet the preset expansion and split conditions. However, if 990 keys are stored in the current data node, since 1010>1000, it meets the preset expansion and split conditions. When the data node is full, the space will be expanded through the preset expansion and split mechanism, see Figure 5 As shown in the figure, the expansion and splitting mechanism is similar to the insertion of the B+ tree data structure. If the array can be inserted, it will be inserted. If the array is full, it will be split and expanded.

[0096] 1. Node expansion mechanism: Assume that the total number of keys of a data node is n, such as 1010 above. When expanding a data node containing n keys, allocate a new gap array with a size of n / dl, where dl=0.6 (this is just an example and can be adjusted as needed). Then modify the original linear regression model according to this ratio.

[0097] 2. Node splitting mechanism: In order to split a data node into two parts, this scheme assigns the array value to two new data nodes, so that each new data node is responsible for half of the space of the original data node. The splitting mechanism is similar to that of the B+ tree. Note that by limiting the size of all internal nodes to a power of 2, it is always possible to split the node in a way that preserves the dividing line (i.e., bisection), so there is no need to retrain any model below the split internal node.

[0098] Assume n=1010, n / dl=1683.3, rounded up to 1684, which means the new array can hold 1684 keys, 684 more than the existing 1000. The 1684 can be divided into two parts, either from large to small or from small to large, and the result is the same, that is, the first 842 keys are assigned to one data node, and the remaining 842 keys are assigned to another data node.

[0099] In addition, it is also necessary to modify the parameters in the model index based on the preset value dl, such as a and b in the linear regression model y=ax+b, to obtain a new linear regression model. This step essentially updates a to 0.6a and b to 0.6b, a new linear regression model. Then recalculate the new mapping position of each key in the array to store the data record or pointer of the key in the original data node in the new data node. For example, the original data node is Q, and the new data nodes are W and E. In this case, the pointer or data record of the key in the original data node Q is stored in W and E respectively. The data in W and E are independent of each other, and the original data node Q is deleted.

[0100] It should be noted that the data node can be split first or later, depending on the actual situation. After the split, the data node needs to be re-split, because the original location is data node Q, but at this time after the split, it is necessary to determine whether the final data node is W or E. After determining the data node, perform the data insertion operation. As described above, the data node stores a model index, mapping a key to a position and an array with gaps. This array can store a specific record, such as a specific content, or it can be a pointer to a variable-size record. The pointer points to the specific content in the memory, and the required content can be obtained through the pointer.

[0101] Therefore, for data insertion operations, the model index can be used to determine the projection position of the key mapping. When the data records are stored in the array, the data to be inserted can be inserted into the projection position in the array. When the pointer is stored in the array, the pointer corresponding to the projection position can be determined first, and then the data to be inserted can be inserted into the specific memory location pointed to by the pointer based on the pointer.

[0102] The method provided by the above embodiment can efficiently utilize the characteristics of the cumulative distribution function and combine different types of model indexes to achieve fast and accurate data query and insertion. In particular, through the node expansion and splitting mechanism, the load balancing of data nodes is ensured, and the performance bottleneck caused by the overload of a single data node is avoided. In addition, in the case of data node expansion and splitting, by dynamically adjusting the parameters of the model index, the high efficiency of data insertion and query is guaranteed, which significantly improves the overall performance of the system.

[0103] See also Figure 6 , shows a schematic flow chart of an optional index processing method according to an embodiment of the present invention, comprising the following steps:

[0104] S601: Using a preset loss function, calculate a current loss value of a current tree structure; wherein the current tree structure includes internal nodes and data nodes, and the internal nodes include a root node;

[0105] S602: Use a preset tuner to adjust parameters in the model index, and use the model index with updated parameters to process all internal nodes and data nodes in the tree structure to obtain a new tree structure;

[0106] S603: Using a preset loss function, calculate a new loss value of the new tree structure;

[0107] S604: In response to the new loss value being smaller than the current loss value, the current tree structure is replaced with the new tree structure.

[0108] In the above implementation, time complexity: the number of times the algorithm is executed is the time complexity of the algorithm, which quantitatively describes the time it takes to run the algorithm. Training model: is one of the key indicators for machine learning model optimization, which can help evaluate the performance of the model and provide feedback signals to know the optimization direction of the model.

[0109] For steps S601 to S604, the complexity of query and insertion: both query and insertion must be The traversal is completed with the time complexity of . In the data node, the worst case search is O(logn). The data node model can perfectly predict the location of the key, and the search time is O(1). For insertion, the complexity is O(logn), and the best case is O(1). Here n is the amount of data to be searched. For example, a total of 8 data need to be searched. The worst case is that each time it is necessary to determine which child node the data is in through the internal nodes until the data is finally found, so it takes a maximum of 8 times; the best case is that the data is found through one judgment, so it only needs to be searched once. The same is true for insertion, so I will not go into details.

[0110] In this solution, the model indexes of different internal nodes and data nodes will change, such as 0.6a and 0.6b for data nodes above. However, this has a regular change, that is, it will only change when expansion and splitting are required. For internal nodes, the model index change is similar to the node concept of neural network. The nodes are similar at the beginning, but each node will change as training progresses.

[0111] Training model: The model predicts average search and insertion time based on two simple statistics tracked at each data node: 1. The average number of search iterations 2. The average number of shifts for insertion, such as abd, after inserting data c, changes to acbd, indicating that b shifts once, d shifts once, a total of two shifts, or bd shifts once as a whole. Among them, search performance is directly related to 1., while insertion performance is related to 1. and 2., so the model uses two statistics: 1. The depth of the data node being traversed, 2. The total size of all internal nodes and data node metadata (i.e. the memory size occupied). These two statistics represent the cost of traversal, that is, deeper data nodes require more pointer lookups, and more data nodes will slow down the speed of traversal to the data nodes.

[0112] It should be noted that the purpose of this solution is not to calculate the specific search time and insertion time, but to compare with the current tree structure by calculating the number of search iterations and the average number of displacements. Because the higher the number of iterations, the slower the search; the more operations of inserting data shifts, the slower the insertion. Therefore, a certain calculation method is needed to evaluate the current tree structure and the optimized tree structure. The evaluation is that the lower the average number of search iterations and the average number of insertion displacements, the better the performance, and the faster the query and insertion will be; optimization is to adjust the cumulative distribution function towards this goal. The more accurate the cumulative distribution function prediction, the lower the average number of search iterations and the average number of insertion displacements. Specifically, the cost function can be set as in machine learning to continuously train and iterate the model.

[0113] Assume that there is a tree structure for storing character data and you want to optimize its query and insert performance. You will achieve this goal by following the steps below:

[0114] 1. Get the historical processing records and get the depth of each historical processing record traversed to the data node; the depth is the level from the root node to the gap array, such as Figure 1 As shown, there are 4 levels from the root node to the final array. Assume that there are the following historical processing records:

[0115]

[0116] 2. Get the memory size occupied by all internal nodes and data nodes in the tree structure; internal nodes include the root node. Assume that the memory size occupied by each node is as follows: root node: 10 bytes, data node A: 5 bytes, data node B: 5 bytes, data node C: 5 bytes, data node D: 5 bytes, so the total memory size occupied = 10 + 5 + 5 + 5 +5 = 30 bytes.

[0117] 3. For the historical processing records with the processing type of insert, get the number of shifts and then get the average number of shifts. From the historical records, we can see that the number of shifts when inserting c is 2. Therefore, the average number of shifts is 2.

[0118] 4. Based on one or more of depth, memory size, and average number of shifts, use the preset loss function to calculate the current loss value. Assume that the loss function is: [ \text{Loss}=w_1 \times \text{average depth} + w_2 \times \text{memory size} + w_3 \times \text{average number of shifts} ], and assume that the weights are ( w_1=0.5), ( w_2=0.3 ), and ( w_3=0.2 ). The average depth is calculated to be (1 + 1 + 1 + 1 + 2) / 5=1.2, the memory size is 30 bytes, and the average number of shifts is 2. Therefore, the current loss value is: [ \text{Loss}_{\text{original}}=0.5 \times1.2 + 0.3 \times 30 + 0.2 \times 2=0.6 + 9 + 0.4=10 ]

[0119] Use the preset tuner to adjust the parameters in the model index, use the model index with updated parameters to process all internal nodes and data nodes in the tree structure, obtain the new tree structure, and then obtain the new memory size occupied by all internal nodes and data nodes in the new tree structure. Assume that some parameters are adjusted, such as changing the structure of the tree, the memory size occupied by each node remains unchanged, but the structure is more compact. Use the new tree structure to process the pending keys in the historical processing records to obtain the new depth, and for the historical processing records with the processing type of insert, obtain the number of shifts, and then obtain the new average number of shifts. Reprocess the historical records:

[0120]

[0121] New average depth = (1 + 1 + 1 + 1 + 1) / 5 = 1. New average number of shifts = 0. Based on one or more of the new depth, the new memory size, and the new average number of shifts, a new loss value is calculated using a preset loss function. New loss value: [ \text{Loss}_{\text{new}}=0.5 \times 1 + 0.3 \times 30 + 0.2 \times 0=0.5 + 9+ 0=9.5 ]. In response to the new loss value being less than the current loss value, the current tree structure is replaced with the new tree structure. Since the new loss value of 9.5 is less than the current loss value of 10, it is decided to replace the current tree structure with the new tree structure. Through the above steps, the tree structure is successfully optimized, the average cost of query and insertion is reduced, and the overall performance is improved.

[0122] In actual operation, we use C++ to implement model indexing and conduct experiments. We find that in the case of read-only, the throughput of the model is 4.1 times higher than that of B+Tree, and the size of the index file is reduced by 800 times; in the case of read-write, the throughput of the model is 4.0 times higher than that of B+Tree, and the size of the index file is reduced by 2000 times. The experimental results show that this solution is always better than the traditional B+Tree within the read-write range of the database, and can achieve high performance of the index and low memory usage.

[0123] The method provided in the above embodiment significantly reduces the height of the tree structure by dynamically adjusting the model index parameters and optimizing the tree structure, thereby improving the performance of query and insertion operations and greatly reducing the size of the index file, thereby achieving higher throughput and lower memory usage in read and write operations than traditional B+Tree, thereby improving the overall performance and efficiency of the system.

[0124] This solution designs a model index that replaces the standard database index with a hierarchical structure of a machine learning model, which can greatly improve performance and reduce memory usage when dynamically querying and inserting data in the database. The model index has the following advantages:

[0125] 1. Improve data retrieval efficiency: By efficiently utilizing the characteristics of the cumulative distribution function and combining different types of model indexes, fast and accurate data query and insertion can be achieved.

[0126] 2. Reduce database IO costs: Reduce the storage space occupied by indexes, thereby reducing database IO costs.

[0127] 3. Reduce the storage space occupied by the index: The size of the index file is significantly reduced. Experimental results show that the size of the index file is reduced by 800 times in the read-only case and by 2000 times in the read-write case.

[0128] 4. No need to consume a lot of cost to maintain the index: By dynamically adjusting the model index parameters and optimizing the tree structure, the cost of maintaining the index is reduced.

[0129] In particular, the node expansion and splitting mechanism ensures the load balancing of data nodes and avoids the performance bottleneck caused by the overload of a single data node. In the case of data node expansion and splitting, the model index parameters are dynamically adjusted to ensure the high efficiency of data insertion and query, significantly improving the overall performance of the system.

[0130] See also Figure 7 , which shows a schematic diagram of main modules of an index processing device 700 provided by an embodiment of the present invention, including:

[0131] Receiving module 701, used for receiving keys to be processed;

[0132] The node processing module 702 is used to determine the projection position of the key in the root node using the model index, determine the pointer corresponding to the projection position from the array of pointers to child nodes in the root node, and then determine the child node according to the pointer;

[0133] In response to the type of the child node being an internal node, repeating the above operations of determining the projection position, the pointer, and the child node until the type of the child node determined according to the pointer is a data node;

[0134] The insert module 703 is used for, in response to the type of the child node being a data node, judging whether the data node meets the preset expansion and splitting condition when the processing type is insertion, and if the judgment result is that the data node meets the condition, calling the preset node expansion and splitting strategy to perform the expansion and splitting operation on the data node;

[0135] After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or the pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

[0136] The implementation device of the present invention also includes a query module, which is used to:

[0137] When the processing type is query, the model index is used to determine the projection position of the key, and the data record corresponding to the projection position is read from the gap array and returned, or the pointer corresponding to the projection position is read from the gap array, and then the data record is read from the corresponding storage location in the memory according to the pointer and returned.

[0138] In the implementation device of the present invention, the insertion module 703 is used to:

[0139] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0140] In response to the total number of keys being greater than a preset threshold, it is determined that the data node meets a preset expansion split condition.

[0141] In the implementation device of the present invention, the insertion module 703 is used to:

[0142] Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys;

[0143] Divide the total number of keys by a preset value to obtain a new number of keys, split the new number of keys into two parts in order, and distribute them to two new data nodes; and

[0144] Update the parameters in the model index of the data node based on the preset value to obtain a new model index, and then determine the new mapping position of each key according to the new model index;

[0145] Store the pointer or data record of each key in the data node in the new data node according to the new mapping position.

[0146] The implementation device of the present invention also includes a model updating module, which is used to:

[0147] Using a preset loss function, calculate the current loss value of the current tree structure; wherein the current tree structure includes internal nodes and data nodes, and the internal nodes include the root node;

[0148] Use the preset tuner to adjust the parameters in the model index, use the model index with updated parameters to process all internal nodes and data nodes in the tree structure, and obtain a new tree structure;

[0149] Using a preset loss function, a new loss value of the new tree structure is calculated, and in response to the new loss value being less than the current loss value, the current tree structure is replaced with the new tree structure.

[0150] In the implementation device of the present invention, the parameters for calculating the loss value include one or more of the following:

[0151] Get the historical processing records and get the depth of each historical processing record traversed to the data node; the depth is the level from the root node to the gap array;

[0152] Get the memory size occupied by all internal nodes and data nodes;

[0153] For the historical processing records whose processing type is insert, the number of shifts is obtained, and then the average number of shifts is obtained.

[0154] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0155] Figure 8 An exemplary system architecture 800 to which embodiments of the present invention may be applied is shown, including terminal devices 801 , 802 , 803 , a network 804 and a server 805 (only an example).

[0156] Terminal devices 801, 802, 803 may be various electronic devices with display screens and supporting web browsing, and may be installed with various communication client applications. Users may use terminal devices 801, 802, 803 to interact with server 805 via network 804 to receive or send messages, etc.

[0157] The network 804 is used to provide a medium for communication links between the terminal devices 801, 802, 803 and the server 805. The network 804 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0158] The server 805 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 805, and accordingly, the device is generally set in the server 805.

[0159] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0160] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0161] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0162] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0163] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0164] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a receiving module, a node determination module, and an insertion module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the insertion module may also be described as an "insertion processing module".

[0167] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above index processing methods.

[0168] The computer program product of the present invention comprises a computer program, and when the computer program is executed by a processor, the index processing method in the embodiment of the present invention is implemented.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An index processing method, characterized in that: include: Receive the key to be processed, determine the projection position of the key in the root node using the model index, determine the pointer corresponding to the projection position from the array of pointers to child nodes in the root node, and then determine the child node according to the pointer; In response to the type of the child node being an internal node, repeating the above operations of determining the projection position, the pointer, and the child node until the type of the child node determined according to the pointer is a data node; In response to the type of the child node being a data node, when the processing type is insert, determining whether the data node meets a preset expansion and splitting condition, and when the determination result is that the condition is met, calling a preset node expansion and splitting strategy to perform an expansion and splitting operation on the data node; After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or the pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

2. The method according to claim 1, characterized in that: The method further comprises: When the processing type is query, the model index is used to determine the projection position of the key, and the data record corresponding to the projection position is read from the gap array and returned, or the pointer corresponding to the projection position is read from the gap array, and then the data record is read from the corresponding storage location in the memory according to the pointer and returned.

3. The method according to claim 1, characterized in that The determining whether the data node meets the preset expansion and splitting conditions includes: Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys; In response to the total number of keys being greater than a preset threshold, it is determined that the data node meets a preset expansion split condition.

4. The method according to claim 1 or 3, characterized in that: The calling of the preset node expansion and splitting strategy to perform the expansion and splitting operation on the data node includes: Get the number of keys currently stored in the data node, determine the number of keys to be inserted, and get the total number of keys; Divide the total number of keys by a preset value to obtain a new number of keys, split the new number of keys into two parts in order, and distribute them to two new data nodes; and Update the parameters in the model index of the data node based on the preset value to obtain a new model index, and then determine the new mapping position of each key according to the new model index; Store the pointer or data record of each key in the data node in the new data node according to the new mapping position.

5. The method according to claim 1, characterized in that The method further comprises: Using a preset loss function, calculate the current loss value of the current tree structure; wherein the current tree structure includes internal nodes and data nodes, and the internal nodes include the root node; Use the preset tuner to adjust the parameters in the model index, use the model index with updated parameters to process all internal nodes and data nodes in the tree structure, and obtain a new tree structure; Using a preset loss function, a new loss value of the new tree structure is calculated, and in response to the new loss value being less than the current loss value, the current tree structure is replaced with the new tree structure.

6. The method according to claim 5, characterized in that The parameters for calculating the loss value include one or more of the following: Get the historical processing records and get the depth of each historical processing record traversed to the data node; the depth is the level from the root node to the gap array; Get the memory size occupied by all internal nodes and data nodes; For the historical processing records whose processing type is insert, the number of shifts is obtained, and then the average number of shifts is obtained.

7. An index processing device, characterized in that: include: A receiving module, used for receiving keys to be processed; A node processing module, used to determine the projection position of the key in the root node using the model index, determine the pointer corresponding to the projection position from the array of the root node containing pointers to the child nodes, and then determine the child node according to the pointer; In response to the type of the child node being an internal node, repeating the above operations of determining the projection position, the pointer, and the child node until the type of the child node determined according to the pointer is a data node; The insert module is used for, in response to the type of the child node being a data node, judging whether the data node meets the preset expansion and splitting condition when the processing type is insertion, and if the judgment result is that the condition is met, calling the preset node expansion and splitting strategy to perform the expansion and splitting operation on the data node; After the expansion split is completed or the result is judged to be non-compliant, the projection position of the key is determined using the model index, and the data to be inserted is inserted into the projection position in the gap array, or the pointer corresponding to the projection position is read from the gap array to insert the data to be inserted into the memory storage position pointed to by the pointer.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images