An indexing method, apparatus, and electronic device based on a learned index.
By introducing buffers and Bloom filters into the learning index to manage data modifications, the performance issues of the learning index during data deletion and modification are resolved, achieving efficient data management and improved query performance.
Patent Information
- Application Number
- CN202211510152.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-29
- Publication Date
- 2026-03-03
- Estimated Expiration
- 2042-11-29
AI Technical Summary
Current learning indexes cannot handle data deletion and modification, especially learning indexes with B-tree caches, which are difficult to maintain after data changes, impacting performance.
The system uses a buffer to store data modification operations, separates insert and delete columns to manage new and deleted data, uses a Bloom filter to quickly determine data existence, combines a learned index for query verification, and regularly trains the model to update the index.
It effectively handles data addition, deletion, and modification operations, improves the performance of learning indexes, reduces the maintenance requirements of B-tree structures, and enhances query efficiency and space utilization.
Smart Images
Figure CN115712616B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database indexing technology, and in particular to an indexing method, apparatus, and electronic device based on a learned index. Background Technology
[0002] An index is a technique used in databases to speed up queries, much like a table of contents in a book, facilitating quick data retrieval. Indexes are typically stored using structures like B+ trees, and as the amount of data grows, the index itself will consume more and more storage space.
[0003] Learning indexes, based on AI technology, differ significantly from traditional indexes. They train a neural network model to predict the approximate location of a key. The advantages of learning indexes include: 1. They predict key locations through computation, thus minimizing storage consumption. 2. They uncover internal relationships within the data, which traditional indexes completely ignore. For data with explicit or implicit patterns, learning indexes can achieve extremely high performance with minimal space usage.
[0004] However, a major drawback of learned indexes is that they are difficult to maintain after data changes, because they require retraining the neural network model with the updated data, which can significantly impact performance. Summary of the Invention
[0005] The technical problem to be solved by the present invention is that current learning indexes (including learning indexes with B-tree caches) cannot handle data deletion and data modification. In view of this, the present invention provides an indexing method, apparatus and electronic device based on learning indexes.
[0006] The technical solution adopted in this invention is that the indexing method based on a learned index includes:
[0007] In response to data modification operations in the database, the corresponding modified data is stored in a pre-configured buffer outside the original database;
[0008] In response to a query request for data in the database, the database is queried based on a pre-configured learning index, and the query is verified using the buffer.
[0009] In one implementation, the modification operation includes at least one of adding, deleting, and replacing data, and the buffer is pre-configured with insertion and deletion sequences.
[0010] In one implementation, when the modification operation is an add operation, the added data is stored in the insert column; when the modification operation is a delete operation, the deleted data is stored in the delete column; when the modification operation is a replace operation, the added data is stored in the insert column, and the deleted data is stored in the delete column.
[0011] In one implementation, querying the database based on a pre-configured learned index and performing query verification using the buffer includes:
[0012] When the learning index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it returns null; if the corresponding data does not exist in the deletion column, the data has not been deleted and the data is returned.
[0013] If the learning index does not find the corresponding data, the corresponding data is searched in the insertion column. If the corresponding data exists in the insertion column, the data is returned; otherwise, an empty string is returned.
[0014] In one implementation, querying the database based on a pre-configured learned index and performing query verification using the buffer includes:
[0015] When the learning index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it searches for the corresponding data in the insertion column. If the data exists in the insertion column, it returns the data. If the data does not exist in the insertion column, it returns null. If the corresponding data does not exist in the deletion column, it returns the data.
[0016] When the learning index does not find the corresponding data, the corresponding data is found in the insertion column. If the corresponding data exists in the insertion column, the corresponding data is found in the deletion column. If the data exists in the deletion column, an empty string is returned. If the data does not exist in the deletion column, the data is returned. If the corresponding data does not exist in the insertion column, an empty string is returned.
[0017] When the same data is found in both the inserted column and the deleted column, that data is deleted from both columns simultaneously.
[0018] In one implementation, a Bloom filter is used to perform query processing on the inserted and deleted columns.
[0019] In one embodiment, the method further includes:
[0020] When the data in the buffer reaches a pre-configured storage threshold, the model of the learning index is trained using the data in the buffer.
[0021] Clear all data in the current buffer.
[0022] Another aspect of the present invention provides an indexing device based on a learned index, comprising:
[0023] The buffer module is configured to store the modified data in a pre-configured buffer outside the original database in response to data modification operations in the database.
[0024] The query module is configured to respond to a query request for data in the database by querying the database based on a pre-configured learned index and using the buffer to verify the query.
[0025] In one embodiment, the indexing device based on a learned index further includes:
[0026] The training module is configured as follows:
[0027] When the data in the buffer reaches a pre-configured storage threshold, the model of the learning index is trained using the data in the buffer.
[0028] Clear all data in the current buffer.
[0029] Another aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the learning-based indexing method as described in any of the preceding claims.
[0030] Another aspect of the present invention provides a computer storage medium storing a computer program that, when executed by a processor, implements the steps of the learning-based indexing method as described in any of the preceding claims.
[0031] By adopting the above technical solution, the present invention has at least the following advantages:
[0032] 1. The indexing method based on learning indexes described in this invention can be used to address situations where learning indexes (including learning indexes with B-tree caches) cannot handle data deletion and data modification, and can handle data addition, deletion, and modification.
[0033] 2. This invention does not require maintaining a complex structure like a B-tree. Instead, it uses a simple list to store data and a Bloom filter to quickly determine if data exists. This significantly improves performance compared to traditional learned indexes (including learned indexes with B-tree caches). Attached Figure Description
[0034] Figure 1 This is a flowchart of an indexing method for a learning index according to an embodiment of the present invention;
[0035] Figure 2 This is a schematic diagram of a modification operation on a buffer according to an embodiment of the present invention;
[0036] Figure 3 This is a schematic diagram of the data query process according to an embodiment of the present invention;
[0037] Figure 4 This is a flowchart of an indexing method for another learning index according to an embodiment of the present invention;
[0038] Figure 5 This is a schematic diagram of the indexing device structure of a learning index according to an embodiment of the present invention;
[0039] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0040] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.
[0041] In the accompanying drawings, the thickness, size, and shape of the objects have been slightly exaggerated for ease of illustration. The drawings are for illustrative purposes only and are not drawn to scale.
[0042] It should also be understood that the terms "comprising," "including," "having," "containing," and / or "comprising," when used in this specification, indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire listed feature, not individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to an example or illustration.
[0043] As used herein, the terms “basically,” “approximately,” and similar terms are used as terms of approximation rather than terms of degree, and are intended to describe inherent biases in measured or calculated values that will be recognized by those skilled in the art.
[0044] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms (e.g., those defined in common dictionaries) shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and shall not be interpreted in an idealized or overly formal sense unless expressly so specified herein.
[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0046] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily to be strictly followed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.
[0047] The first embodiment of the present invention provides an indexing method based on a learned index, such as... Figure 1 As shown (can also be combined) Figure 4 The process includes the following specific steps:
[0048] Step S1: In response to a data modification operation in the database, the corresponding modified data is stored in a pre-configured buffer outside the original database.
[0049] In this embodiment, the modification operation may include at least one of the following: adding, deleting, and replacing data.
[0050] For reference Figure 2 In this embodiment, database modification operations can be performed by configuring a buffer and inserting lists (addList) and deleting lists (delList) within the buffer. Changed data will be cached in these two lists. These two lists have a simple structure, require no maintenance, and can be accessed quickly.
[0051] Specifically, this embodiment can add a buffer to the learning index. For newly added or modified data, the learning index model will not be trained immediately, but will be stored in this buffer first. Only when the data stored in the buffer reaches a threshold will the data in the buffer be used to train the learning index model, and then the buffer will be cleared.
[0052] Furthermore, the buffer maintains two lists (or arrays), AddList and DelList. If a piece of data is inserted, it is placed in AddList; if a piece of data is deleted, it is placed in DelList; if a piece of data is updated, the old value before the update is placed in DelList, and the new value after the update is placed in AddList.
[0053] It's important to note that these two arrays (AddList and DelList) are independent of the index model and stored in memory, allowing for high-speed access. The data in these arrays doesn't need sorting and accumulates over time, resulting in very low maintenance costs. Furthermore, most current similar technologies can only handle key insertions and deletions; they cannot handle key updates. The setup in this embodiment allows for handling key updates because the action of updating a key is equivalent to deleting the old key (adding it to DelList) and inserting the new key (adding it to AddList). This covers all types of key operations in the database.
[0054] Step S2: In response to a query request for data in the database, the database is queried based on a pre-configured learning index, and the query is verified using a buffer.
[0055] In this embodiment, please refer to Figure 3 If data is found in the learning index, then it is necessary to check if the data exists in DelList. If the data exists in DelList, it means that the data has been deleted, and an empty string is returned; if the data does not exist in DelList, it means that the data has not been deleted, and the data is returned as the result.
[0056] If no data is found in the learning index, it needs to be searched again in AddList. If the data exists in AddList, it is returned as the result; otherwise, it is returned empty.
[0057] In some implementations, to achieve more efficient data querying, Bloom filters can be configured in both the addList and delList.
[0058] In some implementations, to minimize the impact of potential errors on the indexing results, the method may further include:
[0059] When the learning index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it searches for the corresponding data in the insertion column. If the data exists in the insertion column, it returns the data. If the data does not exist in the insertion column, it returns null. If the corresponding data does not exist in the deletion column, it returns the data.
[0060] When the learning index does not find the corresponding data, the corresponding data is found in the insertion column. If the corresponding data exists in the insertion column, the corresponding data is found in the deletion column. If the data exists in the deletion column, an empty string is returned. If the data does not exist in the deletion column, the data is returned. If the corresponding data does not exist in the insertion column, an empty string is returned.
[0061] Furthermore, the final state of a key can be determined using AddList and DelList. If a key appears in AddList but not in DelList, it means that the key was newly inserted. If a key appears in DelList but not in AddList, it means that the key was newly deleted.
[0062] In other words, when a key appears in both AddList and DelList, regardless of the order in which they appear, the original state of the key in the index remains unchanged. That is, if the key exists in the index, and then appears in both AddList and DelList, the key in the index will still exist; if the key does not exist in the index, and then appears in both AddList and DelList, the key in the index will still not exist.
[0063] Therefore, when the same data exists in both the insert and delete columns, since the data result remains unchanged after deletion, the data can be deleted simultaneously in both columns. In this case, AddList and DelList will be compressed, reducing space consumption and improving query efficiency.
[0064] Furthermore, it can be combined with Figure 4 In this embodiment, when the data in the buffer reaches the pre-configured storage threshold, the data in the buffer is used to train the model of the learning index, and all data in the current buffer is cleared.
[0065] It is understandable that the specific configuration of the storage threshold can be manually selected within a reasonable range based on the actual application situation, and this article will not limit it further.
[0066] In summary, by adopting the above technical solution, this embodiment has at least the following advantages:
[0067] 1. The indexing method based on learning indexes described in this invention can be used to address situations where learning indexes (including learning indexes with B-tree caches) cannot handle data deletion and data modification, and can handle data addition, deletion, and modification.
[0068] 2. This invention does not require maintaining a complex structure like a B-tree. Instead, it uses a simple list to store data and a Bloom filter to quickly determine if data exists. This significantly improves performance compared to traditional learned indexes (including learned indexes with B-tree caches).
[0069] The second embodiment of the present invention, corresponding to the first embodiment, introduces an indexing device based on a learned index, such as... Figure 5 As shown, it includes the following components:
[0070] The buffer module is configured to store the modified data in a pre-configured buffer outside the original database in response to data modification operations in the database.
[0071] The query module is configured to respond to query requests for data in the database by querying the database based on a pre-configured learned index and using a buffer for query validation.
[0072] In some embodiments of this example, the indexing device based on the learned index further includes:
[0073] The training module is configured to: when the data in the buffer reaches a pre-configured storage threshold, use the data in the buffer to train the model of the learning index, and clear all data in the current buffer.
[0074] In this embodiment, the modification operation includes at least one of adding, deleting and replacing data, and the buffer is pre-configured with insertion and deletion columns.
[0075] In some embodiments of this example, the buffer module is further configured as follows:
[0076] When a modification operation is changed to an insert operation, the added data is stored in the insert column;
[0077] When the modification operation is changed to a deletion operation, the deleted data is stored in the deletion list;
[0078] When the modification operation is changed to a replacement operation, the added data is stored in the insert column, and the deleted data is stored in the delete column.
[0079] In some embodiments of this example, the query module is further configured as follows:
[0080] When the learned index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it returns null; if the corresponding data does not exist in the deletion column, the data has not been deleted and the data is returned.
[0081] When the learning index does not find the corresponding data, the corresponding data is searched in the insertion column. If the corresponding data exists in the insertion column, the data is returned; otherwise, null is returned.
[0082] In some embodiments of this example, the query module is further configured to perform query processing on the inserted and deleted columns using a Bloom filter.
[0083] A third embodiment of the present invention provides an electronic device, such as... Figure 6 As shown, it can be understood as a physical device, including a processor and a memory storing processor-executable instructions, which, when executed by the processor, perform the following operations:
[0084] Step S1: In response to a data modification operation in the database, the corresponding modified data is stored in a pre-configured buffer outside the original database.
[0085] Step S2: In response to a query request for data in the database, the database is queried based on a pre-configured learning index, and the query is verified using a buffer.
[0086] In the fourth embodiment of the present invention, the indexing method based on a learning index follows the same process as the first, second, or third embodiments. The difference lies in the engineering implementation: this embodiment can be implemented using software plus necessary general-purpose hardware platforms. While hardware implementation is also possible, the former is often a preferred method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk), including several instructions to cause a device to execute the method described in the embodiments of the present invention.
[0087] Through the description of specific embodiments, a more in-depth and specific understanding should be gained of the technical means and effects adopted by the present invention to achieve the intended purpose. However, the accompanying drawings are only provided for reference and illustration and are not intended to limit the present invention.
Claims
1. An indexing method based on a learned index, characterized in that, include: In response to data modification operations in the database, the corresponding modified data is stored in a pre-configured buffer outside the original database; In response to a query request for data in the database, the database is queried based on a pre-configured learning index, and the query is verified using the buffer. The querying of the database based on the pre-configured learned index and the query verification using the buffer include: When the learned index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it returns null; if the corresponding data does not exist in the deletion column, the data has not been deleted and the data is returned. When the learning index does not find the corresponding data, the corresponding data is searched in the insertion column. If the corresponding data exists in the insertion column, the data is returned; otherwise, null is returned.
2. The indexing method based on a learned index according to claim 1, characterized in that, The modification operation includes at least one of adding, deleting and replacing data, and the buffer is pre-configured with insertion and deletion columns.
3. The indexing method based on a learned index according to claim 2, characterized in that, When the modification operation is an addition operation, the added data is stored in the insertion column; When the modification operation is a deletion operation, the deleted data is stored in the deletion array; When the modification operation is a replacement operation, the added data is stored in the insertion column, and the deleted data is stored in the deletion column.
4. The indexing method based on a learned index according to claim 1, characterized in that, The querying of the database based on the pre-configured learned index and the query verification using the buffer include: When the learning index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it searches for the corresponding data in the insertion column. If the data exists in the insertion column, it returns the data. If the data does not exist in the insertion column, it returns null. If the corresponding data does not exist in the deletion column, it returns the data. When the learning index does not find the corresponding data, the corresponding data is found in the insertion column. If the corresponding data exists in the insertion column, the corresponding data is found in the deletion column. If the data exists in the deletion column, an empty string is returned. If the data does not exist in the deletion column, the data is returned. If the corresponding data does not exist in the insertion column, an empty string is returned. When the same data is found in both the inserted column and the deleted column, that data is deleted from both columns simultaneously.
5. The indexing method based on a learned index according to claim 4, characterized in that, The inserted and deleted columns are queried using a Bloom filter.
6. The indexing method based on a learned index according to claim 4, characterized in that, The method further includes: When the data in the buffer reaches a pre-configured storage threshold, the model of the learning index is trained using the data in the buffer. Clear all data in the current buffer.
7. An indexing device based on a learned index, characterized in that, include: The buffer module is configured to store the modified data in a pre-configured buffer outside the original database in response to data modification operations in the database. The query module is configured to respond to a query request for data in the database, query the database based on a pre-configured learned index, and perform query verification using the buffer. The query module is further configured as follows: When the learned index finds the corresponding data, it searches for the corresponding data in the deletion column. If the corresponding data exists in the deletion column, it returns null; if the corresponding data does not exist in the deletion column, the data has not been deleted and the data is returned. When the learning index does not find the corresponding data, the corresponding data is searched in the insertion column. If the corresponding data exists in the insertion column, the data is returned; otherwise, null is returned.
8. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the indexing method based on a learning index as described in any one of claims 1 to 6.
9. A computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the indexing method based on a learning index as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Double-annular-array data storage method and intelligent gateway
CN104035884A
Database index optimization method and readable storage medium
CN113392089A