Database system data query method and device based on index cache and medium

Abstract

The invention discloses a database system data query method and device based on index cache and a medium, and relates to the field of data query, the method comprises the following steps: writing specified data into a database according to a data writing request, and generating an operation record; updating the index cache based on the priority corresponding to the operation record in the memory queue; based on a preset time period, performing persistence processing on the index cache to generate an index file; and determining a corresponding query path according to whether a data table corresponding to the data query request has an index or not, and performing data query in an index cache or an index file based on the query path and whether the index cache is hit or not to obtain a query result. The problem that write-in performance and data consistency are difficult to consider at the same time is effectively solved, and the requirements of various application scenes with high requirements for database performance and data accuracy are met.

Description

Technical Field

[0001] The present application relates to the field of data query, and in particular to a method, device and medium for querying data in a database system based on an index cache. Background Art

[0002] In traditional database operations, when data is inserted or updated into an indexed table, the database immediately updates the corresponding index as the data is written. While this synchronization ensures real-time consistency between data and indexes, because index updates are performed simultaneously with data writes, the database must expend significant time and resources to maintain the consistency of the index structure each time data is written. For example, with a B-tree index, each insertion or update of data may cause nodes to split or merge. These operations not only involve disk I / O but also require complex index structure adjustments, increasing the time overhead of the write operation. In high-concurrency write scenarios, as the number of concurrent writes increases, this synchronization can create a serious performance bottleneck, resulting in slow data writes and low system throughput.

[0003] To improve write performance, some database systems attempt to use an asynchronous approach to handle index generation. That is, when data is written, the index is not generated first, and a dedicated thread or process is used to asynchronously build the index later. However, in asynchronous index generation technology, although the write performance problem is solved, since the query operation may be executed before the index update is completed, it is difficult to guarantee the accuracy of the query results, which brings data consistency issues. For example, after the data is inserted, the query operation may not be able to obtain the newly inserted data because the index is not updated in time, or after the data is updated, the query still returns the old index information, resulting in data inconsistency. The query results are difficult to reflect the latest data status, affecting the availability of the database system. This inconsistency is difficult to accept in application scenarios with high requirements for data accuracy (such as financial transaction systems, medical data management systems, etc.). Summary of the Invention

[0004] To solve the above problems, this application proposes a database system data query method based on index cache, including:

[0005] According to the data writing request, the specified data is written into the database and an operation record is generated;

[0006] Update the index cache based on the priority corresponding to the operation record in the memory queue;

[0007] Based on a preset time period, the index cache is persistently processed to generate an index file;

[0008] According to whether there is an index in the data table corresponding to the data query request, the corresponding query path is determined, and based on the query path and whether the index cache is hit, data query is performed in the index cache or the index file to obtain the query result.

[0009] In one example, according to a data write request, specified data is written into a database and an operation record is generated, specifically including:

[0010] Receive a data write request and perform a validity check on the specified data corresponding to the data write request;

[0011] Determining that the legality check passes, writing the specified data into the corresponding data table in the database;

[0012] Generate an operation record for the write operation corresponding to the data write request; the operation record includes at least an operation type;

[0013] The operation record is persisted by appending.

[0014] In one example, updating the index cache based on the priority corresponding to the operation record in the memory queue specifically includes:

[0015] Determine a memory queue established based on priority; the fields of the memory queue include operation type, data row primary key, priority, and transaction ID;

[0016] Determining the priority of the operation record based on the operation type weight, data heat weight, and urgency mark corresponding to the operation record;

[0017] Based on the priority, encapsulate the operation record and add it to the memory queue, and control the queue capacity of the memory queue according to the priority;

[0018] The index cache is updated according to the memory queue.

[0019] In one example, based on the priority, encapsulating the operation record and adding it to the memory queue, and controlling the queue capacity of the memory queue according to the priority, specifically includes:

[0020] Recording the cache update time after the operation record is added to the memory queue;

[0021] Based on the cache update time, the priority of the operation record is increased until it reaches the upper limit of the priority corresponding to the operation record;

[0022] When the queue capacity of the memory queue reaches a preset capacity upper limit, the operation records whose priority is lower than the preset priority are discarded;

[0023] There are multiple preset capacity upper limits, each of which is set with a corresponding preset priority, and as the preset capacity upper limit increases, the preset priority of discarding increases.

[0024] In one example, updating the index cache based on the priority corresponding to the operation record in the memory queue specifically includes:

[0025] Generate a thread in the memory queue through the index, read the corresponding operation record according to the priority corresponding to each operation record, and parse the operation record to obtain the parsing result;

[0026] Determine the index structure used by the index cache;

[0027] Based on the index structure and according to the parsing result, locating the corresponding memory index node;

[0028] Based on the memory index node, updating the index cache according to the operation type corresponding to the operation record included in the parsing result;

[0029] Determine that the index cache update is complete and return a success flag.

[0030] In one example, based on a preset time period, performing persistence processing on the index cache to generate an index file specifically includes:

[0031] Trigger persistence processing based on a preset time period;

[0032] In the index cache, filter and obtain the index data that needs to be persisted;

[0033] arranging and optimizing the index data;

[0034] The index data is written into an index file, and based on a format requirement corresponding to the index file, the index file is written into a storage device.

[0035] In one example, the method further includes:

[0036] determining, within the preset time period, that the number of the data write requests reaches a first preset number;

[0037] Determining whether the current situation meets a preset situation; wherein the preset situation includes: the number of newly added or modified index nodes in the index cache reaches a second preset number, and the current CPU utilization is lower than a preset threshold;

[0038] During the preset time period, a persistence process is triggered.

[0039] In one example, according to whether an index exists in a data table corresponding to a data query request, a corresponding query path is determined, and based on the query path and whether the index cache is hit, a data query is performed in the index cache or the index file to obtain a query result, specifically including:

[0040] Receive a data query request, determine whether a corresponding index structure exists in the data table corresponding to the data query request, and determine the query condition included in the data query request;

[0041] If the index structure exists, performing a search operation in the index cache to attempt to locate a memory index node that meets the query condition in the index cache based on the index structure;

[0042] If the attempt is successful, then the corresponding data row is obtained in the data table as the query result according to the memory index node;

[0043] If the attempt fails, performing a search operation in the index file to read the index information according to the reading rule corresponding to the index file, and obtaining the corresponding data row in the data table as the query result according to the index information;

[0044] If no index structure exists, a traversal query is performed on the data table to obtain data rows that meet the query conditions as query results.

[0045] On the other hand, the present application also proposes a database system data query device based on index cache, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the database system data query method based on index cache as described in any of the above examples.

[0049] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: a database system data query method based on index cache as described in any of the above examples.

[0050] The database system data query method based on index cache proposed in this application can bring the following

[0051] Beneficial effects:

[0052] 1. Use transaction logs to record data write operations, and use the index generation thread to asynchronously update the index cache from the transaction log to decouple data writes from index updates, improving write performance while ensuring data consistency.

[0053] By separating data writing and index updates and using transaction logs to record data writing operations, the data writing module can focus on quickly persisting data to the data table, avoiding the blocking of write operations by index updates, thereby significantly improving data writing performance.

[0054] In high-concurrency write scenarios, the data write module can quickly respond to a large number of write requests without waiting for index updates to complete, thereby improving system throughput.

[0055] 2. The introduction of the index cache module enables queries to prioritize index information from memory, reducing disk I / O operations. Because memory access is much faster than disk access, query response time is significantly reduced. Furthermore, the index generation thread asynchronously updates the index cache via the transaction log, ensuring that the data in the index cache promptly reflects the latest data changes, thereby ensuring the accuracy and consistency of query results.

[0056] 3. Dynamically schedule index update tasks through priority queues, prioritizing high-frequency data and urgent operations to ensure query performance for core businesses. In high-concurrency scenarios, the system automatically identifies high-frequency data and urgent operations, prioritizing updates to key indexes to avoid query performance degradation caused by index delays.

[0057] 4. Build an index persistence module to regularly persist the data in the index cache to disk to ensure the security and persistence of the index data. Even if a memory failure occurs in the system, the index data can be restored through the index file on the disk, ensuring the reliability of the system and the security of the data.

[0058] 5. Develop a query process based on index cache-first lookups to optimize query performance and ensure that query results reflect the latest data status. This effectively solves the problem of balancing write performance and data consistency, meeting the needs of various application scenarios with high requirements for database performance and data accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1Schematic diagram of a flow chart of a method for querying data in a database system based on index cache in an embodiment of the present application;

[0061] Figure 2 This is a schematic diagram of the overall architecture of a data query system in one scenario in an embodiment of the present application;

[0062] Figure 3 This is a schematic diagram of the processing flow of the data writing module in one scenario in an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of a processing flow of an index cache module in one scenario in an embodiment of the present application;

[0064] Figure 5 This is a schematic diagram of the processing flow of the index persistence module in one scenario in an embodiment of the present application;

[0065] Figure 6 This is a schematic diagram of the processing flow of the query module in one scenario in an embodiment of the present application;

[0066] Figure 7 Schematic diagram of a database system data query device based on index cache in an embodiment of the present application. DETAILED DESCRIPTION

[0067] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0068] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0069] To address the aforementioned issues, a version number mechanism can be introduced into the database. Each data entry in the table is assigned a version number, which increments when the data changes. The index also records the version number of the corresponding data, and comparing the two versions determines whether the index is up to date.

[0070] As for the version number mechanism, although the timeliness of the index can be judged to a certain extent through version number comparison, this solution still has certain limitations. For example, in high-concurrency and complex operation scenarios, frequent version number updates and comparison operations will consume a lot of CPU and memory resources. In addition, for complex database operations (such as batch updates, transaction nesting, etc.), the maintenance and management of version numbers becomes more complicated, and it is easy to have inconsistent version numbers, which in turn affects data consistency. For example, in a transaction that includes a large amount of data updates, the version numbers of a large amount of data need to be updated. At the same time, the version numbers of each data and its index need to be compared during the query, which will increase the burden on the system and reduce the overall performance of the system. At the same time, in complex situations such as transaction nesting, the transmission and management of version numbers are prone to errors, further affecting data consistency.

[0071] Based on this, Figure 1 As shown, an embodiment of the present application provides a database system data query method based on index cache, which is applied in a data query system.

[0072] like Figure 2 As shown, the data query system mainly includes a data writing module, a transaction log module, a priority queue module, an index cache module, an index persistence module and a query module.

[0073] like Figure 1 As shown, the method includes:

[0074] S101: According to a data writing request, the specified data is written into the database and an operation record is generated.

[0075] The data write module is primarily responsible for receiving external data write requests and performing validation to ensure that the specified data complies with the table's structure and constraints. Once validation passes, the corresponding data is written directly into the table and an operation record (including transaction_id, operation type, row primary key, affected rows, etc.) is generated and recorded in the transaction log module.

[0076] The transaction log module uses an append-write method to persist operation records to the transaction log file on the disk to ensure that even if a system failure occurs, the data status can be restored through the transaction log and the operation records can be pushed to the priority queue module.

[0077] Of course, if the data write request is an urgent operation triggered by an administrator (eg, manual index rebuilding), a corresponding urgent flag urgent=true may be generated.

[0078] Specifically, if Figure 3As shown, the external system initiates a data write request to the database, and the request carries the data to be written, which is referred to as designated data herein.

[0079] After receiving an external data write request, the data write module performs a validity check on the data corresponding to the request. This validity check can be based on the pre-defined structure and constraints of the data table, strictly checking the requested data's field type, length, value range, and whether it complies with unique constraints and foreign key constraints.

[0080] If the validity check is passed, the specified data will be written into the corresponding data table of the database. The data writing module writes the specified data that has passed the validity check into the data table of the database to complete the initial storage of the data.

[0081] For each write operation corresponding to a data write request, an operation record is generated; the operation record includes at least the operation type. The data write module encapsulates the detailed information of the write operation, including the operation type (insert, update, or delete), the primary key of the affected data row, and the data values before and after the change, into an operation record and sends it to the transaction log module.

[0082] The transaction log module persists the operation records in the transaction log file on the disk by appending the operation records.

[0083] If the data is illegal and fails the validity check, an error is returned to the external system, indicating that there is a problem with the data and that it needs to be corrected before the request can be initiated again.

[0084] S102: updating the index cache based on the priority corresponding to the operation record in the memory queue.

[0085] The priority queue module receives operation records from the transaction log module and sorts them according to priority rules. Operations marked with the urgent=true flag are prioritized. Regular operations are prioritized based on operation type (e.g., insert > update > delete) and data popularity (generally prioritizing high-frequency data).

[0086] Specifically, a memory queue established based on priority is determined; the fields of the memory queue include operation type, data row primary key, priority, and transaction ID.

[0087] Use a priority-based memory queue (for example, Java-based PriorityBlockingQueue or C++-based priority_queue). Queue elements are encapsulated operation records containing the following fields:

[0088] operation_type indicates the operation type, which can include insert, update, and delete; row_key indicates the primary key of the data row; priority indicates the priority, which is dynamically calculated by the system. For example, insert operations have higher priority than update operations, and high-frequency data rows have higher priority; transaction_id indicates the transaction ID, which is used for atomicity verification.

[0089] Determine the priority of the operation record based on the operation type weight, data heat weight, and urgency mark corresponding to the operation record.

[0090] In the priority calculation rules, the operation type weight can be set to 100 for inserts, 80 for updates, and 50 for deletes. Of course, the specific weight values can be set based on needs. Data popularity weights are also set, based on query frequency statistics over the past seven days, with high-frequency data rows receiving an additional 30 points.

[0091] Of course, if it carries an urgent mark, that is, the operation containing urgent=true (for example, the index rebuild manually triggered by the administrator, in which case the urgent field of the transaction log is true) is directly placed on the top and set to the highest priority.

[0092] Based on the priority, the operation records are encapsulated and added to the memory queue, and the queue capacity of the memory queue is controlled according to the priority.

[0093] During encapsulation, operations are encapsulated at regular intervals, in descending order of priority. Queue capacity is controlled, for example, with a maximum capacity of max_queue_size = 10000. When the queue is full, the lowest priority operation is automatically discarded and logged. A queue_remaining() interface is provided for the index generation thread to dynamically adjust processing speed.

[0094] At this point, the index cache can be updated based on the memory queue.

[0095] Furthermore, considering that the priority calculation rules are relatively fixed, in some extreme cases, low-priority operations may remain in the queue for a long time, causing index update delays and affecting the consistency of some data.

[0096] Based on this, the cache update time after the operation record is added to the memory queue is recorded. This time can be recorded in seconds.

[0097] Based on the cache update time, the priority of the operation record is increased until it reaches the upper limit of the corresponding priority of the operation record. For example, the priority score increases by 10% for every 1 second of waiting, up to 150% of the initial value. This allows operation records that have been in the memory queue for a long time to be prioritized, but not too high a priority that affects the priority processing of other more important operation records.

[0098] In addition, when the memory queue reaches the preset capacity limit, operation records with a priority lower than the preset priority are discarded. As mentioned above, when the queue is full, the lowest priority operation is automatically discarded and logged. However, this method is still prone to problems in extreme cases when the amount of operation records is too large.

[0099] Therefore, multiple preset capacity upper limits are set, and each preset capacity upper limit is set with a corresponding preset priority, and as the preset capacity upper limit increases, the discarded preset priority increases, that is, as the queue capacity increases, operation records with higher priorities are gradually discarded. For example, the preset capacity upper limits are set to 10,000, 15,000, and 20,000, respectively. When the queue capacity reaches 10,000, operation records with a priority score lower than 30 are discarded. If at this time, the queue capacity still rises and reaches 15,000, operation records with a priority score lower than 40 are discarded. If at this time, the queue capacity continues to rise and reaches 20,000, operation records with a priority score lower than 50 are discarded. Of course, the 50 can be used as the set upper limit value, and operation records with higher priorities are no longer discarded, thereby preventing useful operation records from being discarded.

[0100] The index cache module includes an index generation thread that periodically retrieves the highest-priority operations from the priority queue. It then locates the in-memory index node (e.g., a node in a B-tree structure) by parsing the operation type (including insert, update, and delete). It then performs index updates, inserting new key-value pairs, modifying node values, or merging or splitting nodes.

[0101] The index generation thread reads new operation records from the priority queue. When an insert or update operation record is read, the corresponding index is updated in the index cache module based on the data rows and index definitions involved in the operation. The index cache module uses an index structure (such as a hash table or tree structure) to store index information, enabling fast index lookups and updates. For example, if a B-tree index structure is implemented in the index cache, node insertion, deletion, and update operations can be performed using the in-memory tree structure.

[0102] Specifically, if Figure 4As shown, the index cache module generates a thread in the memory queue through the index, reads the corresponding operation record according to the priority corresponding to each operation record, and parses the operation record to obtain the parsing result.

[0103] The index generation thread retrieves the highest-priority operation record from the memory queue and updates the index cache, otherwise skipping the operation. The index generation thread accesses the priority queue module at a fixed frequency (for example, 10 times per second) to retrieve the highest-priority operation record. The operation records in the queue are sorted according to the priority rules in the priority queue module.

[0104] Parse the retrieved operation records to extract the operation type (including insert, update, delete, etc.), data row primary key and index column information.

[0105] Determine the index structure used by the index cache; based on the index structure and according to the parsing result, locate the corresponding memory index node (also referred to as index node for short).

[0106] In the index cache module, the corresponding index node is located based on the index columns and primary key information. For example, when the index structure is a B-tree index, the target node is located through binary search.

[0107] Based on the memory index node, the index cache is updated according to the operation type corresponding to the operation record included in the parsing result.

[0108] For insert operations, a new (index column value, row pointer) key-value pair is inserted into the index node. If the node is full, a split is triggered. For update operations, the index column value of the corresponding primary key in the index node is modified. For delete operations, the key-value pair of the specified primary key is removed from the index node. If the node is half empty, a merge is triggered.

[0109] Confirm that the index cache update is complete and return a success flag. After the update is complete, return a success flag to the priority queue module, and the queue automatically deletes the operation record. If the update fails, record the error log and retain the operation record for retrying.

[0110] S103: Based on a preset time period, perform persistence processing on the index cache.

[0111] The index persistence module organizes and optimizes index cache data based on a time period (for example, every 5 minutes), merges partially empty nodes, and writes them to disk index files to ensure data recovery after a memory failure.

[0112] To prevent data loss in the index cache due to memory failures, the index persistence module periodically persists data from the index cache to index files on disk. During persistence, the module first organizes and optimizes the data in the index cache, for example, by merging adjacent index nodes to reduce disk I / O. It then writes the index data to disk according to the disk index file format requirements.

[0113] Specifically, if Figure 5 As shown, based on a preset time period, the persistence process is triggered. According to the preset time period (for example, 5 minutes), the index persistence module is triggered to start working.

[0114] In the index cache, the index data that needs to be persisted is filtered out. The index persistence module filters the index data that needs to be persisted to disk from the index cache.

[0115] Organize and optimize index data. For example, merge adjacent index nodes with less data to reduce the number of subsequent disk I / O operations and improve persistence efficiency.

[0116] Write the index data to the index file, and then write the index file to the storage device based on the format requirements of the index file. According to the specific format requirements of the disk index file, write the organized and optimized index data to the disk line by line, completing the persistent storage of the index data from memory to disk.

[0117] After the index is persisted, wait for the next cycle to execute the persistence operation again.

[0118] Furthermore, considering that the time in this case is fixed, when faced with relatively frequent data access, once a failure occurs, data loss is likely to occur.

[0119] Based on this, if it is determined that the number of data write requests reaches a first preset number within a preset time period, it means that the current access is relatively frequent and system data failure is likely to occur.

[0120] Determine whether the current situation meets the preset conditions; the preset conditions include: the number of newly added or modified index nodes in the index cache reaches a second preset number, and the current CPU utilization is below a preset threshold. For example, if the number of newly added or modified index nodes reaches 1000 and the CPU utilization is below 40%, it indicates that there are a large number of index nodes to be processed and the device has a high level of remaining unused computing resources. In this case, a temporary persistence process is triggered within a preset time period, thereby balancing data security and performance, if the device allows, and reducing the risk of data loss due to failures.

[0121] S104: Determine the corresponding query path based on whether the data table corresponding to the data query request has an index, and based on the query path and whether the index cache is hit, perform data query in the index cache or the index file to obtain the query result.

[0122] After receiving a data query request, the query module checks for the existence of the index structure. If no index structure exists, a full table scan is performed. If an index structure exists, the query module prioritizes searching the index cache module. If a cache hit occurs, the data row is located based on the index node and the query result is returned. If a cache miss occurs, the index is read from the index file, the cache is updated, and the result is returned.

[0123] When a data query request arrives, the query module first checks whether an index structure exists for the data table involved in the query. If an index structure exists, the corresponding index information is preferentially searched from the index cache module. If an index node that meets the query criteria is found in the index cache, the query module retrieves the corresponding data row from the data table based on the row pointer or other positioning information provided by the index node and returns it to the external system. If the required index information is not found in the index cache, the query module reads the index information from the index file on disk.

[0124] Specifically, if Figure 6 As shown, a data query request is received, and it is determined whether a corresponding index structure exists in the data table corresponding to the data query request, and the query condition included in the data query request is determined.

[0125] The external system sends a data query request to the database's query module, which contains information such as query conditions. The query module determines whether the data table involved in the query has an index structure and uses this to determine the subsequent query path.

[0126] If an index structure exists, a search operation is performed in the index cache to attempt to locate a memory index node that meets the query criteria in the index cache based on the index structure.

[0127] The query module enters the index cache module and, based on the query conditions, uses the efficient search algorithm of the memory data structure to try to locate the index node that meets the conditions in the index cache.

[0128] If the attempt is successful, the corresponding data row is retrieved from the data table based on the in-memory index node as the query result. In other words, if a hit is found in the index cache, the data row is retrieved based on the index and the query result is returned. The query module uses the row pointer or other positioning information provided by the index structure to accurately retrieve the corresponding data row from the database data table and return the query result to the external requesting system.

[0129] If the attempt fails, a search operation is performed in the index file to read the index information according to the corresponding reading rules of the index file. Based on the index information, the corresponding data row is retrieved from the data table as the query result. In other words, if there is a miss in the index cache, a search operation is performed from the index file. The query module then turns to the index file on disk and reads the index information according to the reading rules of the disk index file. Based on the index information obtained from the index file, the query module retrieves the data from the data table and returns the final query result to the external requesting system.

[0130] Index information describes the index structure (e.g., B-tree, hash) and attribute metadata, while memory index nodes are the specific data structure units of the index structure in the memory cache. Index information defines how to construct memory index nodes, and memory index nodes are the specific implementation of index information in the cache.

[0131] If no index structure exists, a traversal query is performed on the data table to obtain rows that meet the query criteria as the query results. In this case, a full table scan is performed directly to obtain the rows and return the results. The query module traverses the entire data table, filters out rows that meet the query criteria, and returns the results to the external requesting system.

[0132] 1. Use transaction logs to record data write operations, and use the index generation thread to asynchronously update the index cache from the transaction log to decouple data writes from index updates, improving write performance while ensuring data consistency.

[0133] By separating data writing and index updates and using transaction logs to record data writing operations, the data writing module can focus on quickly persisting data to the data table, avoiding the blocking of write operations by index updates, thereby significantly improving data writing performance.

[0134] In high-concurrency write scenarios, the data write module can quickly respond to a large number of write requests without waiting for index updates to complete, thereby improving system throughput.

[0135] 2. The introduction of the index cache module enables queries to prioritize index information from memory, reducing disk I / O operations. Because memory access is much faster than disk access, query response time is significantly reduced. Furthermore, the index generation thread asynchronously updates the index cache via the transaction log, ensuring that the data in the index cache promptly reflects the latest data changes, thereby ensuring the accuracy and consistency of query results.

[0136] 3. Dynamically schedule index update tasks through priority queues, prioritizing high-frequency data and urgent operations to ensure query performance for core businesses. In high-concurrency scenarios, the system automatically identifies high-frequency data and urgent operations, prioritizing updates to key indexes to avoid query performance degradation caused by index delays.

[0137] 4. Build an index persistence module to regularly persist the data in the index cache to disk to ensure the security and persistence of the index data. Even if a memory failure occurs in the system, the index data can be restored through the index file on the disk, ensuring the reliability of the system and the security of the data.

[0138] 5. Develop a query process based on index cache-first lookups to optimize query performance and ensure that query results reflect the latest data status. This effectively solves the problem of balancing write performance and data consistency, meeting the needs of various application scenarios with high requirements for database performance and data accuracy.

[0139] like Figure 7 As shown, the embodiment of the present application also proposes a database system data query device based on index cache, including:

[0140] at least one processor; and,

[0141] a memory communicatively connected to the at least one processor; wherein,

[0142] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the database system data query method based on index cache as described in any of the above embodiments.

[0143] The embodiment of the present application further proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the index cache-based database system data query method described in any of the above embodiments.

[0144] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0145] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0146] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0152] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0154] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for querying data in a database system based on index cache, characterized in that: include: According to the data writing request, the specified data is written into the database and an operation record is generated; Update the index cache based on the priority corresponding to the operation record in the memory queue; Based on a preset time period, the index cache is persistently processed to generate an index file; According to whether there is an index in the data table corresponding to the data query request, the corresponding query path is determined, and based on the query path and whether the index cache is hit, data query is performed in the index cache or the index file to obtain the query result.

2. The method for querying data in a database system based on index cache according to claim 1, characterized in that: According to the data write request, the specified data is written into the database and an operation record is generated, including: Receive a data write request and perform a validity check on the specified data corresponding to the data write request; Determining that the legality check passes, writing the specified data into the corresponding data table in the database; Generate an operation record for the write operation corresponding to the data write request; the operation record includes at least an operation type; The operation record is persisted by appending.

3. The method for querying data in a database system based on index cache according to claim 1, characterized in that: Based on the priority corresponding to the operation record in the memory queue, the index cache is updated, specifically including: Determine a memory queue established based on priority; the fields of the memory queue include operation type, data row primary key, priority, and transaction ID; Determining the priority of the operation record based on the operation type weight, data heat weight, and urgency mark corresponding to the operation record; Based on the priority, encapsulate the operation record and add it to the memory queue, and control the queue capacity of the memory queue according to the priority; The index cache is updated according to the memory queue.

4. The method for querying data in a database system based on index cache according to claim 3, characterized in that: Based on the priority, encapsulating the operation record and adding it to the memory queue, and controlling the queue capacity of the memory queue according to the priority, specifically including: Recording the cache update time after the operation record is added to the memory queue; Based on the cache update time, the priority of the operation record is increased until it reaches the upper limit of the priority corresponding to the operation record; When the queue capacity of the memory queue reaches a preset capacity upper limit, the operation records whose priority is lower than the preset priority are discarded; There are multiple preset capacity upper limits, each of which is set with a corresponding preset priority, and as the preset capacity upper limit increases, the preset priority of discarding increases.

5. The method for querying data in a database system based on index cache according to claim 1, characterized in that: Based on the priority corresponding to the operation record in the memory queue, the index cache is updated, specifically including: Generate a thread in the memory queue through the index, read the corresponding operation record according to the priority corresponding to each operation record, and parse the operation record to obtain the parsing result; Determine the index structure used by the index cache; Based on the index structure and according to the parsing result, locating the corresponding memory index node; Based on the memory index node, updating the index cache according to the operation type corresponding to the operation record included in the parsing result; Determine that the index cache update is complete and return a success flag.

6. The method for querying data in a database system based on index cache according to claim 1, characterized in that: Based on a preset time period, the index cache is persistently processed to generate an index file, specifically including: Trigger persistence processing based on a preset time period; In the index cache, filter and obtain the index data that needs to be persisted; arranging and optimizing the index data; The index data is written into an index file, and based on a format requirement corresponding to the index file, the index file is written into a storage device.

7. The method for querying data in a database system based on index cache according to claim 6, characterized in that: The method further comprises: determining, within the preset time period, that the number of the data write requests reaches a first preset number; Determining whether the current situation meets a preset situation; wherein the preset situation includes: the number of newly added or modified index nodes in the index cache reaches a second preset number, and the current CPU utilization is lower than a preset threshold; During the preset time period, a persistence process is triggered.

8. The method for querying data in a database system based on index cache according to claim 1, characterized in that: Determine the corresponding query path based on whether an index exists in the data table corresponding to the data query request, and perform a data query in the index cache or the index file based on the query path and whether the index cache is hit to obtain a query result, specifically including: Receive a data query request, determine whether a corresponding index structure exists in the data table corresponding to the data query request, and determine the query condition included in the data query request; If the index structure exists, performing a search operation in the index cache to attempt to locate a memory index node that meets the query condition in the index cache based on the index structure; If the attempt is successful, then the corresponding data row is obtained in the data table as the query result according to the memory index node; If the attempt fails, performing a search operation in the index file to read the index information according to the reading rule corresponding to the index file, and obtaining the corresponding data row in the data table as the query result according to the index information; If no index structure exists, a traversal query is performed on the data table to obtain data rows that meet the query conditions as query results.

9. A database system data query device based on index cache, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the index cache-based database system data query method as described in any one of claims 1 to 8 above.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: the method for querying database system data based on index cache as described in any one of claims 1 to 8 above.

Images