Method for constructing index structure, index data writing, reading method and device

Abstract

The application provides a construction method of an index structure, an index data writing and reading method and device. The construction method comprises grouping basic data, constructing a leaf node corresponding to each group of basic data, and performing linear fitting on the basic data of each leaf node to obtain a linear model; calculating a target storage location of target basic data of the leaf node based on the linear model, and storing the target basic data to the target storage location when the target storage location is empty, or storing the target basic data to a cache node opened corresponding to the target storage location when the target storage location is occupied, to complete the initialization of the leaf node and the cache node under the leaf node; and obtaining a constructed index structure according to an index model of a root node determined in the initialized leaf node. The index structure can better support a concurrent protocol, and the performance of a streaming distribution algorithm for fitting basic data and a retraining strategy are better.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to an index structure construction method, index data writing and reading method and device. BACKGROUND

[0002] Index is one of the main technologies for improving database access performance in database systems. In the era of big data, with the continuous growth of data volume, the problems of traditional indexes (such as B+ tree) are increasingly prominent: (1) The space cost is too high. For example, B+ tree index needs to index the original data with O(n) additional space, which is difficult to tolerate in the big data environment. (2) Each query needs multiple indirect searches. For example, each query in B+ tree needs to access all nodes on the path from the tree root to the leaf node, which limits the search performance of B+ tree to the data size. Since 2018, artificial intelligence as a method of learning data distribution has given birth to a new research direction of "learning index". At present, the design of learning index is still in its infancy and there are many problems. For example, the learning index has the problem of insufficient fitting ability for basic data, and the performance of the streaming distribution algorithm fitting the basic data is poor; the existing learning index retraining design is too idealistic, the retraining frequency is too high, the single retraining time is too long, and it cannot support the update operation, or the index update strategy is not practical; most high-performance learning indexes cannot support the concurrent protocol well, or the performance becomes worse than the traditional index after supporting the concurrent protocol. SUMMARY

[0003] Therefore, the present application aims to provide an index structure construction method, index data writing and reading method and device, which can better support the concurrent protocol, and optimize the streaming distribution algorithm performance and retraining strategy of the learning index fitting the basic data.

[0004] The index structure construction method provided by the present application comprises:

[0005] Obtaining basic data and grouping the basic data to obtain multiple groups of basic data;

[0006] Constructing a leaf node corresponding to each group of basic data, and performing linear fitting on the basic data corresponding to each leaf node to obtain a linear model of the leaf node; wherein each leaf node comprises a plurality of storage locations;

[0007] based on the linear model of the leaf node, calculate a target storage location of target base data of the leaf node, and store the target base data to the target storage location when the target storage location is blank, or store the target base data to a cache node opened corresponding to the target storage location when the target storage location is occupied, to complete initialization of the leaf node and cache nodes under the leaf node;

[0008] construct a root node, and determine an index model of the root node according to key base data in each initialized leaf node, to obtain a constructed index structure.

[0009] In some embodiments, the method for constructing the index structure includes, when the target storage location is blank, storing the target base data to the target storage location, and when the target storage location is occupied, storing the target base data to a cache node opened corresponding to the target storage location, to complete initialization of the leaf node and cache nodes under the leaf node.

[0010] sort the base data of the leaf node according to a preset sorting rule, and repeat the following steps in order of arrangement of the base data until the last base data.

[0011] based on the linear model of the leaf node, calculate a target storage location of target base data of the leaf node;

[0012] determine whether the target storage location of the leaf node is blank;

[0013] if yes, store the target base data to the target storage location;

[0014] if no, open a cache node corresponding to the target storage location, and store the target base data to the cache node, or store the target base data to a cache node corresponding to the target storage location.

[0015] In some embodiments, the method for constructing the index structure further includes:

[0016] after the index structure writes index data, when it is determined that a local index structure of the index structure satisfies a local update condition, acquire index data in the local index structure; the local index structure includes a local index model, and the local index model is an index model of a cache node or a leaf node;

[0017] based on the index data of the local index structure, re-determine a local index model of the local index structure;

[0018] replace the original local index model in the local index structure with the re-determined local index model, and update the local index structure.

[0019] In some embodiments, the method for constructing the index structure, wherein the method comprises:

[0020] inputting the index data of the local index structure into a pre-trained index matching model; wherein the index matching model is obtained by training a neural network model using training data, and the training data comprises sample index data and sample index models;

[0021] determining, by the index matching model, a local index model that matches the index data of the local index structure.

[0022] In some embodiments, the method for constructing the index structure, wherein the method comprises:

[0023] calculating a first accumulation probability based on the insertion time of the index data, the construction or update time of the local index structure, the predicted data accumulation speed, and the number of index data when the construction or update of the local index structure is completed;

[0024] calculating, by a conflict probability model, a conflict probability that the number of conflicting index data in the local index structure caused by the insertion operation is greater than a preset threshold of conflicting index data;

[0025] determining, based on the first accumulation probability and / or the conflict probability, whether the local index structure satisfies the local update condition;

[0026] calculating, by a read probability model, a read update probability that the read frequency in the local index structure caused by the read operation is greater than a preset read frequency;

[0027] calculating a second accumulation probability based on the read time of the index data, the construction or update time of the local index structure, the predicted data accumulation speed, and the number of index data when the construction or update of the local index structure is completed;

[0028] determining, based on the second accumulation probability and / or the read update probability, whether the local index structure satisfies the local update condition. In some embodiments, the method for constructing the index structure further comprises:

[0029] after the index data is written into the index structure, obtaining the index data in the index structure when the index structure satisfies a global update condition and the server state of the index structure satisfies a preset idle condition;

[0030] Updating the leaf nodes, the cache nodes under each leaf node, and the root node of the index structure based on the index data in the index structure.

[0031] In some embodiments, the method for constructing the index structure further comprises:

[0032] Fitting the basis data corresponding to the leaf node with a straight line to obtain a linear model;

[0033] Enlarging the slope of the linear model by a preset multiple to update the linear model, and taking the updated linear model as the linear model of the leaf node.

[0034] In some embodiments, the method for writing index data further comprises:

[0035] Calculating a target leaf node corresponding to the index data to be written by the index model in the root node;

[0036] Processing the index data to be written by the linear model in the target leaf node to determine a predicted position of the index data to be written and a target position in the target leaf node matching the predicted position;

[0037] Determining whether the target position satisfies a preset writing condition; the preset writing condition is that the target position is empty or the data existing in the target position matches the index data to be written;

[0038] If yes, writing the index data to be written into the target position;

[0039] If no, if a cache node corresponding to the target position exists, writing the index data to be written into the cache node; or if a cache node corresponding to the target position does not exist, newly opening a cache node and writing the index data to be written into the newly opened cache node;

[0040] Storing the index data to be written according to the index model in the cache node.

[0041] In some embodiments, the method for reading index data further comprises:

[0042] Calculating a target leaf node corresponding to the index data to be read by the index model in the root node;

[0043] Processing the index data to be read by the linear model in the target leaf node to determine a predicted position of the index data to be read and a target position in the target leaf node matching the predicted position;

[0044] When the target position is empty, it is determined that the search fails;

[0045] When the data stored in the target position matches the index data to be read, it is determined that the search succeeds; when the data stored in the target position does not match the index data to be read, it is determined whether the target position has a corresponding cache node;

[0046] If yes, the index model in the cache node is used to search the index data to be read, and a search result is determined;

[0047] If no, the search fails.

[0048] Embodiments of the present application also provide a construction device of an index structure, and the device comprises:

[0049] An acquisition module is configured to acquire basic data, group the basic data, and obtain multiple groups of basic data;

[0050] A first construction module is configured to construct a leaf node corresponding to each group of basic data, perform linear fitting on the basic data corresponding to each leaf node, and obtain a linear model of the leaf node; each leaf node comprises multiple storage positions;

[0051] A calculation module is configured to calculate a target storage position of target basic data of the leaf node based on the linear model of the leaf node, store the target basic data in the target storage position when the target storage position is empty, and store the target basic data in a cache node opened in the corresponding target storage position when the target storage position is occupied, so as to complete initialization of the leaf node and the cache node below the leaf node;

[0052] A second construction module is configured to construct a root node, determine an index model of the root node according to key basic data in each initialized leaf node, and obtain a constructed index structure.

[0053] The application provides a construction method of an index structure, an index data writing method and device, and an index data reading method and device. When the construction method is used to construct the index structure, the obtained basic data is grouped, data with similar data distribution characteristics is divided into a group, a leaf node corresponding to each group of basic data is constructed, the basic data corresponding to each leaf node is linearly fitted to obtain a linear model of the leaf node, so that the accuracy of the linear model in predicting the position of the data is improved, and the fitting performance of the algorithm is improved. Based on the linear model of the leaf node, a target storage position of target basic data of the leaf node is calculated, and the target basic data is stored in the target storage position when the target storage position is empty, or the target basic data is stored in a cache node opened in the corresponding target storage position when the target storage position is occupied. Finally, a root node is constructed, and an index model of the root node is determined according to the key basic data in each leaf node that has been initialized, so that a constructed index structure is obtained. The learning index constructed has only three layers and is very flat. When new data is inserted, the new data is inserted in each cache node in the leaf node, so that a large amount of new data can be inserted. This structure can ensure that continuous insertion does not increase the height of the tree and keeps the tree flat. In this way, when a node is retrained, the range of influence is small, so that when parallel (concurrent) insertion and searching are performed, the blocking key granularity (range) is small, and the parallel (concurrent) performance of the index is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0055] Figure 1 A basic architecture schematic diagram of a learning index in the prior art is shown.

[0056] Figure 2 A method flowchart of the construction method of the index structure according to the embodiments of the application is shown.

[0057] Figure 3 Linear fitting of the basic data corresponding to the leaf node to obtain the linear model of the leaf node according to the embodiments of the application is shown.

[0058] Figure 4 A schematic diagram of the index structure constructed by the construction method according to the embodiments of the application is shown.

[0059] Figure 5 A method flowchart of another construction method of the index structure according to the embodiments of the application is shown.

[0060] Figure 6 A method flow chart of the index data writing method according to an embodiment of the present application is shown.

[0061] Figure 7 A method flow chart of the index data reading method according to an embodiment of the present application is shown.

[0062] Figure 8 A structure diagram of the index structure construction device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application are only used for the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flow charts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow charts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flow charts or one or more operations can be removed from the flow charts under the guidance of the content of the present application by those skilled in the art.

[0064] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0065] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0066] An index structure is a separate, physical database structure that is a collection of the values in one or several columns of a table and the corresponding list of logical pointers to the data pages in the table that physically contain those values. An index structure provides pointers to the data values stored in the specified columns of a table, ordered according to a user-specified sort order. The database uses the index in much the same way that a user uses an index in a book. An index is saved in the database when the table to which the index is attached is saved, or when the relation graph in which the table resides is saved. In terms of the effect achieved, an index is a data structure that helps a database to efficiently access data, and can be understood as a "quickly searchable data structure that is sorted."

[0067] In recent years, artificial intelligence and machine learning technologies have developed rapidly. Kraska et al. of MIT first designed a learning index based on a machine learning method in 2018. This new technology aims to reduce the space cost and improve the query performance of the index. After that, some more practical and updatable learning indexes such as FITing-tree, PGM-Index, ALEX, XIndex, APEX, LIPP, etc. have been designed, which bring innovations in supporting updates, concurrent operations, hardware-oriented new designs, etc. The core idea of ETC. Learning index is to use a set of models to effectively approximate the cumulative distribution function (CDF) of the data (such as key-value pairs) stored in the sorted array. This set of models can effectively calculate the position of the target key in this array without the need for multiple indirect search actions in the traditional index of tree structure. Therefore, this method has the potential to significantly improve the query performance of the index. These approximate models can be composed of a series of parameters of neural networks or a series of functions (for example, linear functions). Therefore, their space cost is much smaller than that of traditional indexes.

[0068] Index is one of the main techniques to improve database access performance in database systems. In the era of big data, with the continuous growth of data volume, the problems of traditional indexes (such as B+ tree) are increasingly prominent: (1) The space cost is too high. For example, B+ tree index needs to index the original data with the help of O(n) additional space, which is difficult to tolerate in the big data environment. (2) Each query needs multiple indirect searches. For example, each query in B+ tree needs to access all nodes on the path from the tree root to the leaf node, which limits the search performance of B+ tree to the data size. Since 2018, artificial intelligence has emerged as a method to learn data distribution, giving birth to a new research direction of "learning index". At present, the design of learning index is still in its infancy, and there are many problems. For example, learning index has the problem of insufficient fitting ability for basic data, and the performance of the streaming distribution algorithm for fitting basic data is poor; the existing learning index retraining design is too idealistic, the retraining frequency is too high, the time consumption of single retraining is too long, and it cannot support update operation or the index update strategy is not practical; most high-performance learning indexes cannot support concurrent protocols well, or the performance becomes worse than traditional indexes after supporting concurrent protocols.

[0069] For example, a data storage method for improving data write speed and a data read and write request processing method based on the data storage method. The data storage method includes constructing a three-layer tree-shaped data index structure to store data, which includes a model layer, a deconstruction layer, and a data storage layer. The model layer is composed of two layers of learning index models, the first layer of learning index model is an RMI model, and the second layer of learning index model is a plurality of segmented function linear models. The index node array in the deconstruction layer contains one or more index nodes, and each index node contains N data indexes. The data storage layer stores data in the form of data blocks. This method can be executed on an NVM device, and the relatively static deconstruction layer reduces model retraining caused by writing, so that the data storage method can improve the writing performance. As shown in Figure 1 The basic architecture of the previous learning index is shown. This learning index has a high number of internal layers. When a certain internal node needs to be retrained (as shown in the model enclosed in the middle frame), it will affect a very large range of keys (as shown in the lower green frame), making these keys unable to be searched. The range covered by this mode (roughly the range of these keys) cannot be inserted with data, eventually causing the blocking of parallel (concurrent) insertion and search, greatly reducing the parallel (concurrent) performance of the index. Figure 1 Figure 1

[0070] ​​Based on this, the application provides an index structure construction method, index data writing and reading method and device; when the index structure is constructed by the construction method, the obtained basic data is grouped, data with similar data distribution characteristics is divided into a group, and a leaf node corresponding to each group of basic data is constructed, the basic data corresponding to each leaf node is linearly fitted to obtain a linear model of the leaf node, thereby improving the accuracy of the linear model in predicting the position of the data and improving the algorithm fitting performance; based on the linear model of the leaf node, the target storage position of the target basic data of the leaf node is calculated, and when the target storage position is empty, the target basic data is stored in the target storage position, and when the target storage position is occupied, the target basic data is stored in the cache node opened in the corresponding target storage position; finally, a root node is constructed, and the index model of the root node is determined according to the key basic data in each initialized leaf node to obtain the constructed index structure; the constructed learning index has only three layers and is very flat, supports insertion of new data in each cache node in the leaf node, and thus a large amount of new data can be inserted, and this structure can ensure that continuous insertion does not increase the height of the tree and keeps flat; in this way, when a node is retrained, the affected range is small, thereby greatly improving the parallel (concurrent) performance of the index.

[0071] Please refer to Figure 2 , Figure 2 A method flowchart of the index structure construction method described in the embodiments of the application is shown. Specifically, the index structure construction method comprises the following steps S201-S204.

[0072] S201, obtain basic data and group the basic data to obtain multiple groups of basic data;

[0073] S202, construct a leaf node corresponding to each group of basic data, and linearly fit the basic data corresponding to each leaf node to obtain a linear model of the leaf node; each leaf node includes multiple storage positions;

[0074] S203, based on the linear model of the leaf node, calculate the target storage position of the target basic data of the leaf node, and when the target storage position is empty, store the target basic data in the target storage position, and when the target storage position is occupied, store the target basic data in the cache node opened in the corresponding target storage position to complete the initialization of the leaf node and the cache node under the leaf node;

[0075] S204, construct a root node, and determine the index model of the root node according to the key basic data in each initialized leaf node to obtain the constructed index structure.

[0076] The method for constructing the index structure provided in the application groups the obtained basic data, divides data with similar data distribution characteristics into a group, and constructs a leaf node corresponding to each group of basic data. The basic data corresponding to each leaf node is linearly fitted to obtain a linear model of the leaf node, thereby improving the accuracy of the linear model in predicting the position of the data of this type and improving the fitting performance of the algorithm. Based on the linear model of the leaf node, the target storage position of the target basic data of the leaf node is calculated, and the target basic data is stored in the target storage position when the target storage position is empty or in the cache node opened in the corresponding target storage position when the target storage position is occupied. Finally, a root node is constructed, and the index model of the root node is determined according to the key basic data in each leaf node whose initialization is completed to obtain the constructed index structure. The constructed learning index has only three layers and is very flat. When new data is inserted, it is inserted in each cache node in the leaf node, so that a large amount of new data can be inserted. This structure can ensure that continuous insertion does not increase the height of the tree and keeps it flat. In this way, when a node is retrained, the range of influence is small, so that the blocking key granularity (range) is small when inserting and searching in parallel (concurrently), greatly improving the parallel (concurrent) performance of the index.

[0077] In the step S201, basic data is obtained, and the basic data is grouped to obtain a plurality of groups of basic data.

[0078] In the embodiment of the application, the basic data is grouped based on a clustering analysis method. Specifically, the basic data for which the index structure is to be established is analyzed by clustering using K-means or other clustering methods to obtain a plurality of groups of basic data. The number of clusters is the number of second-layer leaf nodes, that is, the number of groups. The points in the same cluster are the basic data in a leaf node.

[0079] The basic data is the data used to construct the index structure. For example, if a company establishes an index structure with the employee name as the key, the employee name data of the company is the basic data.

[0080] In the step S202, a leaf node corresponding to each group of basic data is constructed, and the basic data corresponding to each leaf node is linearly fitted to obtain a linear model of the leaf node. Each leaf node includes a plurality of storage positions.

[0081] Here, the basic data corresponding to different leaf nodes is different, and therefore the linear models obtained by fitting the basic data are different. That is, the linear models of different leaf nodes are different.

[0082] After the basic data is grouped by the clustering analysis method, the basic data in each leaf node has similar data distribution characteristics, a linear model can be better fitted, the error between each technical data and the linear model is smaller, and therefore the fitting capability of the index to the basic data is improved, and the performance of the fitting basic data flow distribution algorithm is improved.

[0083] Specifically, refer to Figure 3 The basic data corresponding to the leaf node is linearly fitted to obtain a linear model of the leaf node, including the following steps S301-S302.

[0084] S301, linearly fitting the basic data corresponding to the leaf node to obtain a linear model.

[0085] S302, expanding the slope of the linear model by a preset multiple to update the linear model, and taking the updated linear model as the linear model of the leaf node.

[0086] Here, after obtaining the linear model of a single leaf node, the steps S301-S302 are repeated to obtain the linear model of each leaf node.

[0087] In the embodiment of the application, the basic data corresponding to a single leaf node is linearly fitted to obtain a linear model of the leaf node, specifically including the following steps:

[0088] Sorting the basic data in the leaf node;

[0089] Using the least square method, the basic data in the leaf node is linearly approximated to obtain a linear model. Specifically, the slope a and the intercept b of the linear function are calculated according to the basic data. That is, the linear model can be represented as y=ax+b. Where y represents the specific storage location of the basic data, and x represents the basic data value;

[0090] The acceptable index size of the leaf node is specified as a reference, and the multiple of the slope a is expanded to a'. That is, the new linear model is updated to y=a'+b, which is the initialized linear model of the leaf node.

[0091] In the step S103, based on the linear model of the leaf node, the target storage location of the target basic data of the leaf node is calculated, and when the target storage location is empty, the target basic data is stored in the target storage location, and when the target storage location is occupied, the target basic data is stored in the cache node opened corresponding to the target storage location, to complete the initialization of the leaf node and the cache node under the leaf node.

[0092] Specifically, the target base data is stored in a target storage location when the target storage location is empty, and the target base data is stored in a buffer node opened in the target storage location when the target storage location is occupied, to complete initialization of the leaf node and the buffer node under the leaf node, including:

[0093] The base data of the leaf node is sorted according to a preset sorting rule, and the following steps are repeated in turn according to the arrangement order of the base data until the last base data;

[0094] Based on the linear model of the leaf node, the target storage location of the target base data of the leaf node is calculated;

[0095] It is judged whether the target storage location of the leaf node is empty;

[0096] If yes, the target base data is stored in the target storage location;

[0097] If no, a buffer node corresponding to the target storage location is opened, and the target base data is stored in the buffer node, or the target base data is stored in the buffer node corresponding to the target storage location.

[0098] That is, after the leaf node is constructed, the base data of the leaf node needs to be stored in the leaf node. The base data can be stored in the storage location of the leaf node or in the buffer node of the next level of the leaf node.

[0099] Here, the storage location of the leaf node, that is, Slot, and the buffer node, also known as buffer node.

[0100] Please refer to Figure 4 , Figure 4 The index structure constructed by the embodiment of the application is shown, and the index structure includes a root node 401, a leaf node 402 and a buffer node 403.

[0101] Here, referring to Figure 4 After the linear model of the leaf node 402 is calculated, the base data of the leaf node 402 is calculated to obtain a storage location y through the linear model. If y1 and y2 calculated by two consecutive base data X1 and X2 exist y1 (integer) = y2 (integer), it indicates that the storage locations of X1 and X2 are the same (denoted as slot 1), and X1 is inserted into slot 1 first. At this time, a new storage space (denoted as buffer 1) needs to be opened under slot 1, and X2 is inserted into the new buffer 1. Figure 4All conflicting points are processed in a similar way, and all buffers newly created in the leaf nodes due to the conflict are initialized.

[0102] Here, y1 calculated from the base data X1 is determined to be stored in slot 1 through y1 (rounding), so that when the slope of the linear model is expanded, not only the storage space of the leaf node can be expanded, but also the error caused by rounding can be avoided.

[0103] The data amount and data distribution characteristics of the base data stored in the buffer are different, and an index model in the buffer can be constructed based on the base data stored in the buffer.

[0104] For example, if the data size stored in the buffer is less than a cache line granularity, the data is stored in an array form; if the data size is greater than a cache line granularity, it is determined which index model to use to initialize according to the distribution of the stored data. The index model includes a linear model, a hash model, a prefix tree model, and the like.

[0105] In the step S204, a root node is constructed, and an index model of the root node is determined according to the key base data in each leaf node whose initialization is completed, to obtain a constructed index structure.

[0106] In the embodiment of the present application, the first base data stored in all leaf nodes is used as the key of the index leaf node, and the root node is initialized by using the keys.

[0107] Specifically, it is determined which index model to use to initialize the root node according to the data distribution characteristics of the keys.

[0108] At this point, a flat index structure is established in the embodiment of the present application, and the index structure has only three layers, in addition to the root node and the leaf node, the buffer node, and no other nodes, so that when a node is retrained, only the data stored in the node is affected, and the range of influence is small; a large amount of new data can be inserted under each slot in the leaf node, and the buffer structure can ensure that continuous insertion does not increase the height of the tree, and the flatness is maintained; such design can block the key granularity (range) when inserting and searching in parallel (concurrently), greatly improving the parallel (concurrent) performance of the index.

[0109] Please refer to Figure 5 In the embodiment of the present application, the method for constructing the index structure further includes the following steps S501-S502.

[0110] S501, after writing the index data in the index structure, judging whether a local index structure of the index structure satisfies a local update condition, and obtaining the index data in the local index structure; the local index structure comprises a local index model, and the local index model is an index model of a cache node or a leaf node;

[0111] S502, re-determining the local index model of the local index structure based on the index data of the local index structure;

[0112] S503, replacing the original local index model in the local index structure with the re-determined local index model, and updating the local index structure.

[0113] That is, in the present application, the re-training can be performed only for a single leaf node or a single buffer node, and only the data stored in the node is affected, and the affected range is small.

[0114] Specifically, judging whether the local index structure of the index structure satisfies the local update condition comprises at least one of the following:

[0115] When the index data is inserted, a first accumulation probability is calculated according to an insertion time of the index data, a construction or update time of the local index structure, a predicted data accumulation speed, and a number of index data when the construction or update of the local index structure is completed;

[0116] A conflict probability is calculated by a conflict probability model, the conflict probability being a probability that a number of conflict index data in the local index structure caused by the insertion operation is greater than a preset conflict index data threshold;

[0117] The local index structure is judged to satisfy the local update condition according to the first accumulation probability and / or the conflict probability;

[0118] Alternatively, when the index data is read, a read update probability is calculated by a read probability model, the read update probability being a probability that a read frequency in the local index structure caused by the read operation is greater than a preset read frequency;

[0119] A second accumulation probability is calculated according to a read time of the index data, the construction or update time of the local index structure, the predicted data accumulation speed, and the number of index data when the construction or update of the local index structure is completed;

[0120] The local index structure is judged to satisfy the local update condition according to the second accumulation probability and / or the read update probability.

[0121] In the embodiments of the present application, whether the local index structure satisfies the local update condition is determined according to the first accumulated probability and / or the conflict probability. The determination can be made independently based on the first accumulated probability and the conflict probability respectively, or the determination can be made jointly based on the first accumulated probability and the conflict probability.

[0122] For example, if the first accumulated probability is greater than a first accumulated probability threshold or the conflict probability is greater than a preset conflict probability, it is determined that the local index structure satisfies the local update condition.

[0123] Alternatively, if the first accumulated probability is greater than the first accumulated probability threshold and the conflict probability is greater than the preset conflict probability, it is determined whether the local index structure satisfies the local update condition.

[0124] In the embodiments of the present application, specifically, whether the local index structure satisfies the local update condition is determined according to whether the product of the first accumulated probability and the conflict probability is greater than a preset first update probability threshold.

[0125] In addition, whether the local index structure satisfies the local update condition is determined according to the second accumulated probability and / or the read update probability. The determination can be made independently based on the second accumulated probability and the read update probability respectively, or the determination can be made jointly based on the second accumulated probability and the read update probability.

[0126] For example, if the second accumulated probability is greater than a second accumulated probability threshold or the read update probability is greater than a preset read update probability threshold, it is determined that the local index structure satisfies the local update condition.

[0127] Alternatively, if the second accumulated probability is greater than the second accumulated probability threshold and the read update probability is greater than the preset read update probability threshold, it is determined whether the local index structure satisfies the local update condition.

[0128] In the embodiments of the present application, it is first determined whether the read update probability is greater than the preset read update probability threshold. If not, it is determined that the local index structure does not satisfy the local update condition. If yes, it is determined whether the second accumulated probability is greater than the second accumulated probability threshold. If yes, it is determined that the local index structure satisfies the local update condition.

[0129] Here, the local update condition corresponding to the local index structure of the index structure is determined according to the historical work load of the local index model of the local index structure. Specifically, a first accumulated probability is calculated when new index data is inserted, or a second accumulated probability is calculated when index data is read. The predicted data accumulation speed is determined according to the historical data accumulation speed of the local index model. When calculating the conflict probability, the preset conflict index data threshold is predicted according to the historical work load. When determining whether the read operation triggers the local index update, the preset read update probability is set according to the historical work load. Index retraining has the following two cases:

[0130] Case one: when the query / write performance of a slot in a leaf node becomes poor, a new local index model that is most suitable for the data distribution characteristics stored in the current buffer is recommended and constructed to replace the old local index model in the original buffer.

[0131] Here, the reason for the poor query / insertion performance is that the index in the buffer does not match the data distribution stored therein.

[0132] Case two: when the number of all stored data in a leaf node reaches a first preset storage threshold; the query performance or the insertion performance of a leaf node becomes poor.

[0133] Here, the main reason for the poor query performance of the leaf node is that most of the stored data is stored in the buffer rather than in the leaf node. The first preset storage threshold is determined according to the service level agreement (SLA) accepted by the user for query delay. The delay tolerance of the user is proportional to the size of the first preset storage threshold.

[0134] Specifically, the local update condition usually includes a first preset storage threshold corresponding to the data stored in the leaf node or the buffer node, and an average error when the leaf node or the buffer node queries the target storage location.

[0135] Specifically, the principle of the local update condition based on probability calculation according to the embodiments of the present application is as follows:

[0136] When new index data is inserted, the insertion adjustment condition needs to satisfy the following two conditions at the same time:

[0137] Condition 1: After the new index data is inserted, the number of data in the local index structure satisfies the following formula (1):

[0138]

[0139] Condition 2: After the new index data is inserted, the number of conflict data in the local index structure satisfies the following formula (2):

[0140]

[0141] Wherein, n.element_num is the number of index data in the local index structure after the new index data is inserted, n.build_num is the number of index data when the local index structure is constructed or updated, n.conflict_num is the number of conflict index data after the new index data is inserted, β is a preset threshold, and α is another preset threshold.

[0142] Here, the meaning of condition 1 is that the data amount in the local index structure is too large, and the local index structure needs to be updated, and the meaning of condition 2 is that the conflict data in the local index structure is too large, and the local index structure needs to be updated.

[0143] Therefore, the traditional way is to maintain a statistical information, and update n.element_num in the statistical information after each data insertion; when each data insertion causes data conflict, update n.conflict_num in the statistical information.

[0144] Each thread needs to maintain these statistical information when inserting data each time. Maintaining statistical information will cause blocking during concurrent insertion, resulting in serious performance decline. Even if the most advanced atomic write operation is used to maintain these statistical information, a large amount of write bandwidth is also needed, resulting in a decline in concurrent performance.

[0145] Therefore, the above condition 1 and condition 2 are converted into calculating the probability of updating the local index structure caused by the insertion of index data in the present application; specifically, the statistical information maintained by each model in the previous index is replaced by a probability value, and only whether the probability value is triggered needs to be determined after each data insertion or reading, without the need to maintain statistical information, thereby improving the concurrent performance of the index.

[0146] According to the above condition 1, a first accumulation probability model for calculating a first accumulation probability is designed.

[0147] In some embodiments, it can be directly judged by the following formula (3):

[0148]

[0149] Wherein, n.speed is the predicted data accumulation speed.

[0150] However, if the predicted data accumulation speed n.speed is too slow, the actual accumulation data speed is too fast, and at this time, the estimation speed will be blocked.

[0151] Based on this, the first accumulated probability p accumulate (p acc ) is converted.

[0152] p acc Ideally, it can increase with the increase of the insertion time of new index data cur time. The first accumulated probability model please refer to the following formula (4):

[0153]

[0154] Here, ε is a constant, used to ensure that p acc is greater than 0.

[0155] Here, cur time in formula (3) and formula (4) is the insertion time of new index data, and n.build_time is the construction or update time of the local index structure, that is, the initial construction time of the local index structure or the last local update time.

[0156] For condition 2 above, the local index structure must be adjusted when this insertion causes a conflict. Based on this, a conflict probability p conflict is set; thus, the expected number of conflicts after triggering adjustment is (the expectation of geometric distribution).

[0157] According to the conflict triggering condition , we get:

[0158]

[0159] Here, α*α′*n.build_num is the preset conflict index data threshold, that is, the predicted number of conflict index data causing update; α′*n.build_num is the number of new index data in the local index structure from the construction or update of the index structure to the insertion of the index data; and α′ is a hyperparameter, which is determined according to the historical load of the local index structure.

[0160] Here, in the traditional way, the statistical information of n.element_num, n.build_num and n.conflict_num needs to be maintained. In the embodiment of the application, it is converted into the number of conflict index data causing update, and the insertion operation is taken as an event, and the conflict probability that the number of index data causing conflict by the insertion operation is greater than the preset conflict index data threshold is calculated through the conflict probability model such as Bernoulli probability model.

[0161] Finally, because condition 1 and condition 2 in the embodiment of the present application need to be satisfied at the same time, the final first update probability of triggering local index structure update is also required to satisfy condition 1 and condition 2 at the same time; assuming that condition 1 and condition 2 correspond to independent condition variables, then the final first update probability p = p acc *p conflict .

[0162] Here, it should be noted that finally it still depends on whether the current insertion of new data causes data duplication to be adjusted, and if no data conflict is caused, the p acc calculated in advance is invalid.

[0163] Based on this, when inserting new data, first calculate the conflict probability p conflict , and then calculate p acc .

[0164] When reading index data, local index structure update is also triggered; the probability setting principle of triggering local index structure update when reading index data is: the node has not been adjusted by write operation for a long time, the node has not been adjusted by read operation for a long time, the accumulation rate of the sub-tree represented by the node (node.speed) is not too slow, if it is too slow, the adjustment probability should be reduced, because only a little data is accumulated since the last write adjustment, the node information that can be relied on when reading, the last adjustment operation is triggered by read or write, and other information is consistent with the information triggered by write.

[0165] Therefore, when reading index data, the accumulation probability p acc should also be triggered, and the probability model of the accumulation probability is consistent with that when writing data, which will not be repeated. At the same time, p conflict does not need to be calculated.

[0166] Here, a read update probability p pq is set, and the corresponding preset read update probability threshold is a small constant value; the embodiment of the present application considers that if p pq is triggered through n times of independent repeated experiments (i.e., Bernoulli probability), it indicates that the read operation is frequent, and the corresponding local index structure needs to be adjusted; therefore, p pq is defined as a hyperparameter, and the preset read update probability threshold needs to be set to a relatively small value according to historical workloads.

[0167] When reading, the read operation is regarded as an event, and the read update probability p pq triggered by the event is calculated.

[0168] When reading, whether to trigger local index structure update is finally determined according to the second accumulation probability p acc when reading data and the read update probability p pq .

[0169] The read operation is finally judged whether the local update condition is met, and the read update probability p is first judged pq whether the preset read update probability threshold is met, and the second accumulated probability p of the read data is judged again when the preset read update probability threshold is met acc whether the second accumulated probability p of the read data is met acc When both are met, the adjustment of the local index operation is triggered. Since the local training of the present application can be specific to a single node, therefore, each leaf node or cache node will maintain a statistical information to support the re-training during concurrent insertion. Specifically, the statistical information can indicate whether the read-write performance of the index model of a node has seriously decreased (the statistical information includes but is not limited to: the total number of data stored under the node, the average error of the node when querying the insertion position, etc.). In this way, each thread needs to maintain these statistical information each time data is inserted. Maintaining statistical information will cause blocking during concurrent insertion, resulting in serious performance degradation. Even if the most advanced atomic write operation is used to maintain these statistical information, it also needs to consume a lot of write bandwidth, resulting in a decrease in concurrent performance.

[0170] In the embodiment of the present application, the local update condition corresponding to the local index structure of the index structure is determined according to the historical workload of the local index model of the local index structure.

[0171] Specifically, we replace the statistical information maintained by each model in the previous index with a probability value, and only need to determine whether to trigger the probability value after each data insertion, without the need to maintain statistical information. Specifically, if it is determined whether to trigger the probability value after each insertion, if triggered, the model needs to be retrained, otherwise, it does not need to be retrained.

[0172] Here, the setting of the probability value of each model is calculated from the historical workload of the model; according to the historical workload, the size of the probability value can be determined. If the historical load indicates that the insertion of data into a model increases in a certain time period, this model is given a larger probability value in this time period, and this model is more likely to be retrained in this time period.

[0173] That is, when data is inserted into a node, the node determines whether to trigger the preset probability value according to a preset calculation rule; if triggered, the node meets the local update condition. The probability value of the node is determined according to the historical workload of the index model of the node.

[0174] In the index structure construction method, the local index model of the local index structure is re-determined based on the index data of the local index structure, comprising:

[0175] inputting index data of the local index structure into a pre-trained index matching model; wherein the index matching model is obtained by training a neural network model based on training data, and the training data includes sample index data and a sample index model;

[0176] determining, by the index matching model, a local index model matching the index data in the local index structure.

[0177] The index matching model is configured to determine a category of the index model corresponding to the node according to the index data of the node. The category of the index model includes a linear model, a hash model, a prefix tree model, and the like.

[0178] Here, the index matching model is not only used for updating the local index structure, but also used to determine the index model of a node when a new cache node is opened and initialized in step S203, and when a root node is constructed and initialized in step S204.

[0179] In the embodiment of the present application, when the number of all stored data in the index is greater than the second preset storage threshold, the adjustment is temporarily not performed, and the global retraining is waited.

[0180] The method for constructing the index structure also includes:

[0181] After the index data is written into the index structure, when the index structure satisfies a global update condition and a server state where the index structure is located satisfies a preset idle condition, the index data in the index structure is obtained.

[0182] Based on the index data in the index structure, the leaf node, the cache node under each leaf node, and the root node of the index structure are updated.

[0183] That is, the global update condition can be that the number of all stored data in the index is greater than the second preset storage threshold; the global update condition can also be that a preset update time point is reached, for example, the index structure is globally updated every three months.

[0184] Based on the index data in the index structure, the leaf node, the cache node under each leaf node, and the root node of the index structure are updated, that is, the leaf node and the cache node under each leaf node of the index structure are updated according to the index structure construction method in steps S201-S204.

[0185] When the index structure needs global update and the server where the index structure is located is idle, the entire index tree is retrained (that is, the insertion and search operations are less), which greatly reduces the influence of global update on the concurrent performance.

[0186] That is, the design of retraining for the existing learning index is too idealistic, the retraining frequency is too high, and the single retraining time is too long. In the embodiment of the application, three granularity retraining rules of cache nodes, leaf nodes and root nodes are designed to ensure that the fine-grained local training frequency is low and the single time is short. The entire index structure is retrained when the server is idle to ensure the flatness and high performance of the index structure. When retraining, the index matching model is used to adaptively recommend a local storage index model, accurately recommend an index structure more suitable for the distribution of local storage data, and greatly improve the performance of the learning index.

[0187] In the embodiment of the application, the index matching model is trained in the following way:

[0188] Step one: Construct a training data set. According to the performance of different real data sets in the past, construct a training set. The training data set includes a plurality of sample index data sets and the optimal sample index model category corresponding to each basic data set.

[0189] Step two: Train the neural network. Train the neural network using the training data set until the neural network meets the preset stopping condition. The preset stopping condition can be that the loss function calculation result of the neural network meets the preset condition, or reaches the preset training times.

[0190] Based on the same inventive concept, the application also provides an index data writing method corresponding to the index structure constructed by the construction method. Since the principle of solving problems of the index data writing method in the embodiment of the application is similar to the above-mentioned construction method, the implementation of the index data writing method can be referred to the implementation of the construction method, and the repeated parts will not be described again.

[0191] Please refer to Figure 6 , Figure 6 The method flowchart of the index data writing method according to the embodiment of the application is shown, specifically, the method includes the following steps S601-S605;

[0192] S601, calculate the target leaf node corresponding to the to-be-written index data through the index model in the root node;

[0193] S602, process the to-be-written index data through the linear model in the target leaf node to determine the predicted position of the to-be-written index data and the target position in the target leaf node matching the predicted position;

[0194] S603, determine whether the target position meets the preset writing condition; the preset writing condition is that the target position is empty or the existing data in the target position matches the to-be-written data;

[0195] S604, if yes, the index data to be written is written into the target position;

[0196] S605, if no, the index data to be written is written into the corresponding cache node in the target position; or, if there is no corresponding cache node in the target position, a cache node is newly created, and the index data to be written is written into the newly created cache node.

[0197] S606, the index data to be written is stored according to the index model in the cache node.

[0198] Here, the index data to be written is written into the target position includes two cases: the target position is empty, and the index data to be written is directly written into the target position; or the data in the target position already exists and matches the data to be written, and the value corresponding to the target position is updated.

[0199] Specifically, the index data writing process based on the constructed index structure is as follows (assuming that the keyword to be written is X2).

[0200] According to the internal index model of the root node of the constructed index structure, the index X2 is written into the target leaf node;

[0201] According to the linear model in the target leaf node, the predicted position (slot) of X2 is calculated.

[0202] Then, if the slot found according to the calculation is empty (empty), X2 is directly inserted into the slot, and the writing is successful; if the value stored in the slot found according to the calculation matches X2, the value corresponding to X2 is updated, and the writing (update) is successful; if the value stored in the slot found according to the calculation does not match X2, there are two cases: the first case, no matching buffer under the current slot, a buffer corresponding to the current slot is created, X2 is inserted into the buffer, and the writing is successful; the second case, there is a buffer corresponding to the current slot, X2 is queried in the buffer (using the index structure in the buffer to find); if it exists, the value corresponding to X2 is updated, and the writing (update) is successful; if it does not exist, X2 is written into the corresponding buffer, and the writing is successful.

[0203] Based on the same inventive concept, the application also provides an index data reading method corresponding to the index structure constructed by the construction method. Since the principle of the index data reading method for solving problems is similar to the construction method, the implementation of the index data reading method can be referred to the implementation of the construction method, and the repeated parts will not be described here.

[0204] Please refer to Figure 7 , Figure 7 The method flowchart of the index data reading method is shown. Specifically, the reading method includes the following steps S701-S706.

[0205] S701, calculating the target leaf node corresponding to the index data to be read through the index model in the root node;

[0206] S702, processing the index data to be read through the linear model in the target leaf node, determining the predicted position of the index data to be read, and the target position in the target leaf node matching the predicted position;

[0207] S703, when the target position is empty, it is determined that the search fails;

[0208] S704, when the data stored in the target position matches the index data to be read, it is determined that the search succeeds; when the data stored in the target position does not match the index data to be read, it is determined whether the target position has a corresponding cache node;

[0209] S705, if yes, searching the index data to be read through the index model and determining the search result;

[0210] S706, if no, the search fails.

[0211] Here, the search result in step S705 is search success or search failure.

[0212] Specifically, the index reading process is as follows (assuming that the key of the search is X1);

[0213] According to the index model in the root node of the constructed index structure, the index X1 is determined to the target leaf node;

[0214] According to the linear model in the target leaf node, the predicted position (slot) of X1 is calculated;

[0215] Then, if the slot found according to the calculation is empty (empty), it is returned that the search fails (X1 does not exist);

[0216] If the value stored in the slot found according to the calculation matches X1, it is returned that the search succeeds;

[0217] If the value stored in the slot found according to the calculation does not match X1, there are two cases; the first case, no match and no corresponding buffer under the current slot, return a lookup failure (X1 does not exist); the second case, no match and there is a corresponding buffer under the current slot, then query X1 in the buffer (use the index structure in the buffer to find); if it exists, return a lookup success; if it does not exist, return a lookup failure.

[0218] Based on the same inventive concept, the application also provides an index structure construction device corresponding to the index structure construction method. Since the principle of the device in the application solves the problem similarly to the index structure construction method described above, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described.

[0219] Please refer to Figure 8 , Figure 8 The structure of the index structure construction device is shown in the structural schematic diagram, and the device includes:

[0220] The acquisition module 801 is configured to acquire the basic data, group the basic data, and obtain a plurality of groups of basic data.

[0221] The first construction module 802 is configured to construct a leaf node corresponding to each group of basic data, and perform a straight line fitting on the basic data corresponding to each leaf node to obtain a linear model of the leaf node. Each leaf node includes a plurality of storage locations.

[0222] The calculation module 803 is configured to calculate a target storage location of target basic data of the leaf node based on the linear model of the leaf node, and store the target basic data in the target storage location when the target storage location is empty, or store the target basic data in a cache node opened corresponding to the target storage location when the target storage location is occupied, to complete the initialization of the leaf node and the cache node under the leaf node.

[0223] The second construction module 804 is configured to construct a root node, and determine an index model of the root node according to the key basic data in each initialized leaf node, to obtain a constructed index structure.

[0224] The index structure construction device provided by the application groups the obtained basic data, divides data with similar data distribution characteristics into a group, and constructs a leaf node corresponding to each group of basic data, performs linear fitting on the basic data corresponding to each leaf node to obtain a linear model of the leaf node, thereby improving the accuracy of the linear model in predicting the position of the data of this type and improving the fitting performance of the algorithm; based on the linear model of the leaf node, the target storage position of the target basic data of the leaf node is calculated, and when the target storage position is empty, the target basic data is stored in the target storage position, and when the target storage position is occupied, the target basic data is stored in the cache node opened corresponding to the target storage position; finally, the root node is constructed, and the index model of the root node is determined according to the key basic data in each leaf node whose initialization is completed, and the constructed index structure is obtained; the learning index constructed has only three layers, is very flat, supports insertion of new data in each cache node in the leaf node, and thus a large amount of new data can be inserted, and this structure can ensure that continuous insertion does not increase the height of the tree and maintains flatness; in this way, when a certain node is retrained, the range of influence is small, thereby when parallel (concurrent) insertion and searching are performed, the blocking key granularity (range) is small, and the parallel (concurrent) performance of the index is greatly improved.

[0225] In some embodiments, the computing module in the index structure construction device stores the target basic data in the target storage position when the target storage position is empty, and stores the target basic data in the cache node opened corresponding to the target storage position when the target storage position is occupied, to complete the initialization of the leaf node and the cache node under the leaf node, and the specific operations include:

[0226] sorting the basic data of the leaf node according to a preset sorting rule, and repeatedly performing the following steps in the order of the arrangement of the basic data until the last basic data;

[0227] calculating the target storage position of the target basic data of the leaf node based on the linear model of the leaf node;

[0228] determining whether the target storage position of the leaf node is empty;

[0229] if yes, storing the target basic data in the target storage position;

[0230] if no, opening a cache node corresponding to the target storage position, and storing the target basic data in the cache node, or storing the target basic data in the cache node corresponding to the target storage position.

[0231] In some embodiments, the index structure construction apparatus further comprises an updating module, configured to, after the index structure writes the index data, acquire the index data in the local index structure when it is determined that the local index structure of the index structure satisfies a local updating condition; the local index structure comprises a local index model, and the local index model is an index model of a cache node or a leaf node;

[0232] re-determine the local index model of the local index structure based on the index data of the local index structure;

[0233] replace the original local index model in the local index structure with the re-determined local index model, and update the local index structure.

[0234] In some embodiments, when re-determining the local index model of the local index structure based on the index data of the local index structure, the updating module in the index structure construction apparatus is specifically configured to:

[0235] input the index data of the local index structure into a pre-trained index matching model; the index matching model is obtained by training a neural network model based on training data, and the training data comprises sample index data and sample index models;

[0236] determine the local index model that matches the index data in the local index structure based on the index matching model.

[0237] In some embodiments, when determining that the local index structure of the index structure satisfies the local updating condition, the updating module in the index structure construction apparatus comprises at least one of the following:

[0238] when inserting the index data, calculate a first accumulation probability based on the insertion time of the index data, the construction or updating time of the local index structure, the predicted data accumulation speed, and the number of index data when the construction or updating of the local index structure is completed;

[0239] calculate, by a conflict probability model, a conflict probability that the number of conflicting index data in the local index structure caused by the insertion operation is greater than a preset conflict index data threshold;

[0240] determine whether the local index structure satisfies the local updating condition based on the first accumulation probability and / or the conflict probability;

[0241] or, when reading the index data, calculate, by a reading probability model, a read updating probability that the read frequency in the local index structure caused by the read operation is greater than a preset read frequency;

[0242] The second accumulation probability is calculated according to a reading time of the index data, a construction or update time of the local index structure, a predicted data accumulation speed, and a quantity of the index data when the construction or update of the local index structure is completed;

[0243] The second accumulation probability and / or the reading update probability are used to determine whether the local index structure satisfies a local update condition. In some embodiments, the update module in the index structure construction device is further configured to, after the index data is written into the index structure, acquire the index data in the index structure when the index structure satisfies a global update condition and a server state in which the index structure is located satisfies a preset idle condition.

[0244] The leaf node, the cache node under each leaf node, and the root node of the index structure are updated based on the index data in the index structure.

[0245] In some embodiments, the first construction module in the index structure construction device is configured to, when performing linear fitting on the basis data corresponding to each leaf node to obtain a linear model of the leaf node, specifically:

[0246] Perform linear fitting on the basis data corresponding to the leaf node to obtain a linear model;

[0247] Enlarge a slope of the linear model by a preset multiple to update the linear model, and use the updated linear model as the linear model of the leaf node.

[0248] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiments, and will not be described herein. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some communication interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0249] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0250] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0251] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0252] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of constructing an index structure, characterized by, The construction method comprises: Obtain basic data, and group the basic data to obtain multiple groups of basic data; Constructing leaf nodes corresponding to each set of basic data, and performing linear fitting on the basic data corresponding to each leaf node to obtain a linear model of the leaf node; wherein each leaf node includes multiple storage locations; Based on the linear model of the leaf node, the target storage location of the target basic data of the leaf node is calculated, and when the target storage location is empty, the target basic data is stored in the target storage location; when the target storage location is occupied, the target basic data is stored in the cache node opened corresponding to the target storage location, so as to complete the initialization of the leaf node and the cache nodes under the leaf node; Construct the root node and determine the index model of the root node based on the key basic data in each initialized leaf node to obtain the constructed index structure; When the target storage location is empty, the target basic data is stored in the target storage location; when the target storage location is occupied, the target basic data is stored in a cache node opened corresponding to the target storage location, so as to complete the initialization of the leaf node and the cache nodes under the leaf node, including: Sort the basic data of the leaf node according to the preset sorting rules, and repeat the following steps in the order of the basic data until the last basic data; Calculate the target storage location of the target basic data of the leaf node based on the linear model of the leaf node; Determine whether the target storage location of the leaf node is empty; If so, storing the target basic data in the target storage location; If not, a cache node is opened corresponding to the target storage location, and the target basic data is stored in the cache node, or the target basic data is stored in the cache node corresponding to the target storage location.

2. The method of claim 1, wherein, The construction method further comprises: After the index data is written into the index structure, when it is determined that a local index structure of the index structure satisfies a local update condition, the index data in the local index structure is obtained; the local index structure includes a local index model, and the local index model is an index model of a cache node or a leaf node; Re-determining a local index model of the local index structure based on the index data of the local index structure; The original local index model in the local index structure is replaced with the re-determined local index model, and the local index structure is updated.

3. The method of claim 2, wherein, Re-determining a local index model of the local index structure based on the index data of the local index structure includes: Inputting the index data of the local index structure into a pre-trained index matching model; wherein the index matching model is obtained by training a neural network model with training data, the training data including sample index data and a sample index model; A local index model that matches the index data in the local index structure is determined through the index matching model.

4. The method of claim 2, wherein, Determining whether the local index structure of the index structure meets the local update condition includes at least one of the following: calculating a first accumulation probability according to an insertion time of the index data, a construction or update time of the local index structure, a predicted data accumulation speed, and a number of index data when the construction or update of the local index structure is completed; calculating a conflict probability of the local index structure according to the conflict probability model, the conflict probability being caused by the insertion operation and being greater than a preset conflict index data threshold; judging whether the local index structure meets a local update condition according to the first accumulation probability and / or the conflict probability; calculating a read update probability of the local index structure according to a read time of the index data, a construction or update time of the local index structure, a predicted data accumulation speed, and a number of index data when the construction or update of the local index structure is completed, the read update probability being caused by the read operation and being greater than a preset read frequency; judging whether the local index structure meets a local update condition according to the second accumulation probability and / or the read update probability. The construction method further comprises:

5. The method of claim 1, wherein, acquiring index data in the index structure when the index structure meets a global update condition and a server state where the index structure is located meets a preset idle condition after the index structure writes the index data; updating leaf nodes, cache nodes under each leaf node, and a root node of the index structure based on the index data in the index structure. performing linear fitting on the basic data corresponding to each leaf node to obtain a linear model of the leaf node, comprising:

6. The method of claim 1, wherein, performing linear fitting on the basic data corresponding to the leaf node to obtain a linear model; enlarging a slope of the linear model by a preset multiple to update the linear model, and taking the updated linear model as the linear model of the leaf node. The method comprises:

7. A method of writing index data, characterized by, calculating a target leaf node corresponding to the to-be-written index data through an index model in the root node; processing the to-be-written index data through a linear model in the target leaf node to determine a predicted position of the to-be-written index data and a target position in the target leaf node that matches the predicted position; judging whether the target position meets a preset write condition; the preset write condition being that the target position is empty or data existing in the target position matches the to-be-written data; if yes, writing the to-be-written index data into the target position; if no, writing the to-be-written index data into a cache node corresponding to the target position if the cache node exists; or, if the cache node does not exist, newly opening up a cache node and writing the to-be-written index data into the newly opened cache node; storing the to-be-written index data according to an index model in the cache node. The method comprises:

8. An index data reading method, characterized by, calculating a target leaf node corresponding to the to-be-read index data through an index model in the root node; processing the to-be-read index data through a linear model in the target leaf node to determine a predicted position of the to-be-read index data and a target position in the target leaf node that matches the predicted position; determining that the search fails when the target position is empty; ​ When the data stored in the target position matches the index data to be read, it is determined that the search is successful; when the data stored in the target position does not match the index data to be read, it is determined whether the target position has a corresponding cache node; If yes, the index model in the cache node is used to search the index data to be read, and a search result is determined; If no, the search fails.

9. An apparatus for constructing an index structure, characterized by The device comprises: An acquisition module is configured to acquire basic data, group the basic data, and obtain multiple groups of basic data; A first construction module is configured to construct a leaf node corresponding to each group of basic data, perform linear fitting on the basic data corresponding to each leaf node, and obtain a linear model of the leaf node; each leaf node comprises multiple storage positions; A calculation module is configured to calculate a target storage position of target basic data of the leaf node based on the linear model of the leaf node, store the target basic data in the target storage position when the target storage position is empty, and store the target basic data in a cache node opened corresponding to the target storage position when the target storage position is occupied, so as to complete initialization of the leaf node and the cache node below the leaf node; A second construction module is configured to construct a root node, determine an index model of the root node according to key basic data in each initialized leaf node, and obtain a constructed index structure; when the target storage position is empty, the target basic data is stored in the target storage position; when the target storage position is occupied, the target basic data is stored in a cache node opened corresponding to the target storage position, so as to complete initialization of the leaf node and the cache node below the leaf node, including: The basic data of the leaf node is sorted according to a preset sorting rule, and the following steps are repeated in turn according to the arrangement order of the basic data until the last basic data; Based on the linear model of the leaf node, a target storage position of target basic data of the leaf node is calculated; It is determined whether the target storage position of the leaf node is empty; If yes, the target basic data is stored in the target storage position; If no, a cache node corresponding to the target storage position is opened, and the target basic data is stored in the cache node, or the target basic data is stored in the cache node corresponding to the target storage position.

Images