Distributed database buffer pool management system and method
By combining AI prediction and memory mapping mechanisms, workload prediction, data preloading, intelligent buffer pool management and dynamic adaptation optimization are adopted, which solves the problem of unstable cache hit rate in traditional database buffer pool management, and improves the database's response ability and query efficiency.
Patent Information
- Application Number
- CN202510511365.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-08-08
AI Technical Summary
The buffer pool management strategy of traditional database management systems lacks the ability to predict future query requests, resulting in unstable cache hit rate and affecting database performance. The operating system's virtual memory management method may lead to latency in querying.
Combining AI prediction and memory mapping mechanisms, efficient database buffer pool management is achieved through workload prediction module, data preload module, intelligent buffer pool management module and dynamic adaptation and feedback optimization module.
It improves the database's response capability in high concurrency scenarios, reduces system I/O latency, and improves query efficiency.
Smart Images
Figure CN120448106A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of databases, in particular to a distributed database buffer pool management system and method. Background Art
[0002] Traditional database management systems use buffer pool technology to optimize disk I / O operations and reduce access latency. However, current buffer pool management strategies, such as LRU (Least Recently Used) or LFU (Least Frequently Used), rely primarily on historical access patterns and lack the ability to predict future changes in query requests. This leads to unstable cache hit rates and impacts database performance. Furthermore, operating systems typically use lazy loading for virtual memory management, which can trigger page faults during queries and increase database access latency.
[0003] In recent years, artificial intelligence (AI) technology has been increasingly used in database management. AI-based workload prediction can analyze query patterns and predict future data access, thereby improving the accuracy of data preloading. Meanwhile, memory mapping mechanisms provide an efficient way to map data files to virtual memory address spaces and automatically load them into physical memory when accessed. However, their default on-demand loading mechanism can still cause access delays.
[0004] How to combine AI prediction and memory mapping mechanism to achieve efficient database buffer pool management is a technical problem that needs to be solved. Summary of the Invention
[0005] The technical task of the present invention is to address the above shortcomings and provide a distributed database buffer pool management system and method to solve the technical problem of how to combine AI prediction and memory mapping mechanism to achieve efficient database buffer pool management.
[0006] In a first aspect, the present invention provides a distributed database buffer pool management system, comprising a workload prediction module, a data preloading module, an intelligent buffer pool management module, and a dynamic adaptation and feedback optimization module;
[0007] The workload prediction module is configured to perform the following operations: collect and store query logs from each database node of the database system, use the query logs as input, and use a trained prediction model to predict the query load arrival of each user in the next time window; generate a list of data shard files required for the next time window based on the prediction results and metadata of the database system; group the list of data shard files according to the nodes where the data shard files are located; and send each group of data shard file lists to a data preloading module of the corresponding database node;
[0008] The data preloading module is used to load all the data of each database node into the virtual memory, obtain the data shard file path according to the received data shard file list, and load the data required for the next time window from the virtual memory to the physical memory based on the memory mapping mechanism;
[0009] The intelligent buffer pool management module is used to lock the data required for the next time window based on the LRU cache elimination strategy of the operating system and the memory usage, and adaptively adjust the size of the time window and the parameters of the prediction model;
[0010] The dynamic adaptation and feedback optimization module is used to optimize the parameters of the prediction model based on the query hit rate in the current time window in units of time windows, and unlock the data shard files based on the required data in the current time window.
[0011] Preferably, the prediction model is a prediction model constructed based on a statistical time series analysis method or a deep learning method, which is used to take query logs as input, analyze the query load arrival of each user in each time window based on the query logs, and predict the query load arrival of each user in the next time window.
[0012] Preferably, the workload prediction module is configured to perform the following operations:
[0013] Collect and store query logs of each database node in the database system;
[0014] Using query logs as input, the trained prediction model is used to predict the query load arrival of each user in the next time window. The query load arrival of each user in the next time window is obtained as the prediction result output;
[0015] Based on the prediction results and the metadata of the database system, a list of data shard files corresponding to the hot database table to be queried in the next time window is determined. The list information of the data shard file list includes the node information of the database nodes where all copies of the data shard files are located and the paths of the corresponding data shard files;
[0016] The data shard file list is used as output, and the data shard file list is grouped according to the node information of the database node where the data shard file is located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node.
[0017] Preferably, the data preloading module is used to perform the following operations:
[0018] Obtain the path of the required data shard file according to the received data shard file list;
[0019] The database system's interface madvise is called, and the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism.
[0020] Preferably, when the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism, the data preloading module loads the data in parallel through a multi-threaded manner.
[0021] Preferably, the intelligent buffer pool management module is configured to perform the following operations:
[0022] L100. Regularly check the memory usage of the database process of the current node. If the usage is less than x% of the maximum available value, execute step L200; otherwise, execute step L300. Here, x% is a database system parameter indicating the database node-level memory usage limit, which can be manually modified by the user.
[0023] L200, call the database system's interface madvise to lock the data required in the next time window; L300, sort the data required in the next time window from high to low priority according to the time dimension from near to far, call the database system's interface mlock to lock the high-priority data and eliminate the low-priority data. At the same time, reduce the size of the time window and adjust the parameters of the prediction model to ensure that in future time windows, there is as little frequent swapping in and out of cached data as possible due to excessive cached data.
[0024] Preferably, the dynamic adaptation and feedback optimization module is configured to perform the following operations:
[0025] The query hit rate is calculated based on the prediction results of the current time window and the actual results of the previous time window, taking the time window as the unit. If the query hit rate is lower than y%, the prediction model is updated and optimized based on the latest query logs collected. y% is a database system parameter that represents the lower limit of the query prediction hit rate.
[0026] Taking the time window as the unit, the data shard files locked in the previous time window are traversed for the current time window. If the data shard file is not in the data shard file list corresponding to the current time window, the database system interface madvise is called to unlock the locked data shard file.
[0027] In a second aspect, the present invention provides a distributed database buffer pool management method, which manages a distributed management system buffer pool by using a distributed database buffer pool management system as described in the first aspect, including the following steps:
[0028] Workload prediction: Query logs from each database node in the database system are collected and stored. Using the query logs as input, the trained prediction model is used to predict the query load arrival of each user in the next time window. Based on the prediction results and the database system's metadata, a list of data shard files required for the next time window is generated. The data shard file lists are grouped according to the nodes where the data shard files are located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node.
[0029] Data preloading: Map all data of each database node to virtual memory. Based on the received data shard file list, the data preloading module obtains the data shard file path and loads the data required for the next time window from virtual memory to physical memory based on the memory mapping mechanism.
[0030] Intelligent buffer pool management: Based on the operating system's LRU cache elimination strategy, it locks the data required for the next time window according to memory usage, and adaptively adjusts the time window size and prediction model parameters;
[0031] Dynamic adaptation and feedback optimization: Optimize the prediction model parameters based on the query hit rate in the current time window and unlock the data shard files based on the data required in the current time window.
[0032] The distributed database buffer pool management system and method of the present invention have the following advantages: using user historical load query prediction to obtain a list of data that may be used in the next time window, using the memory mapping mechanism related interface to load the above data into the physical memory in advance, using the operating system LRU cache elimination strategy combined with the memory mapping locking mechanism to perform intelligent buffer pool management, and performing prediction model detection optimization and memory optimization based on data elimination in units of time windows, avoiding frequent disk I / O reading, reducing system I / O delay, and further improving the responsiveness of the database in high-concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0034] The present invention will be further described below with reference to the accompanying drawings.
[0035] Figure 1 This is a structural block diagram of a distributed database buffer pool management system according to Example 1. DETAILED DESCRIPTION
[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.
[0037] Embodiments of the present invention provide a distributed database buffer pool management system and method for solving the technical problem of how to combine AI prediction and memory mapping mechanism to achieve efficient database buffer pool management.
[0038] Example 1:
[0039] The present invention provides a distributed database buffer pool management system, which comprises a workload prediction module, a data preloading module, an intelligent buffer pool management module and a dynamic adaptation and feedback optimization module.
[0040] The workload prediction module is used to perform the following: collect and store query logs from each database node in the database system, use the query logs as input and the trained prediction model to predict the query load arrival of each user in the next time window, generate the data shard file list required for the next time window based on the prediction results and the metadata of the database system, group the data shard file list according to the node where the data shard file is located, and send each group of data shard file lists to the data preloading module of the corresponding database node.
[0041] In this application, the prediction model is a prediction model constructed based on a statistical time series analysis method or a deep learning method, which is used to take the query log as input, analyze the query load arrival of each user in each time window based on the query log, and predict the query load arrival of each user in the next time window.
[0042] As a specific implementation of the workload prediction module, this module is used to perform the following operations:
[0043] (1) Collect and store query logs of each database node in the database system;
[0044] (2) Using the query log as input, the trained prediction model is used to predict the query load arrival of each user in the next time window, and the query load arrival of each user in the next time window is obtained as the prediction result output;
[0045] (3) Based on the prediction results and the metadata of the database system, a list of data shard files corresponding to the hot database table to be queried in the next time window is determined, wherein the list information of the data shard file list includes the node information of the database nodes where all copies of the data shard files are located and the path of the corresponding data shard files;
[0046] (4) The data shard file list is taken as output, the data shard file list is grouped according to the node information of the database node where the data shard file is located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node.
[0047] The data preloading module is used to obtain the data shard file path according to the received data shard file list, and load the data required for the next time window from the virtual memory to the physical memory based on the memory mapping mechanism.
[0048] As a specific implementation of the data preloading module, this module is used to perform the following operations:
[0049] (1) Obtain the path of the required data segment file according to the received data segment file list;
[0050] (2) Call the database system's interface madvise and load the data required in the next time window from the virtual memory to the physical memory based on the memory mapping mechanism to reduce the data pulling caused by page fault interruptions during query.
[0051] In this embodiment, when the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism, the data preloading module loads the data in parallel through multi-threading to improve the data pulling efficiency.
[0052] The intelligent buffer pool management module is used to lock the data required for the next time window based on the LRU cache elimination strategy of the operating system and the memory usage, and to adaptively adjust the size of the time window and the parameters of the prediction model.
[0053] In this embodiment, the intelligent buffer pool management module is used to perform the following operations:
[0054] L100. Regularly check the memory usage of the database process of the current node. If the usage is less than x% of the maximum available value, execute step L200; otherwise, execute step L300. Here, x% is a database system parameter indicating the database node-level memory usage limit, which can be manually modified by the user.
[0055] L200, call the database system interface madvise to lock the data required in the next time window to prevent the data from being swapped out and improve the memory hit rate;
[0056] L300. Sort the data required in the next time window from high to low priority according to the time dimension, call the database system interface mlock to lock the high-priority data, eliminate the low-priority data, and ensure that the memory usage does not reach the maximum available value. At the same time, reduce the size of the time window and adjust the parameters of the prediction model to ensure that the frequent swapping in and out of cached data due to excessive cached data will occur as little as possible in future time windows.
[0057] The dynamic adaptation and feedback optimization module is used to optimize the parameters of the prediction model based on the query hit rate in the current time window in units of time windows, and unlock the data shard files based on the required data in the current time window.
[0058] As a specific implementation of the dynamic adaptation and feedback optimization module, this module is used to perform the following operations:
[0059] (1) Taking the time window as the unit, the query hit rate is calculated based on the prediction results of the current time window and the actual results of the previous time window. If the query hit rate is lower than y%, the prediction model is updated and optimized based on the latest query logs collected, where y% is a database system parameter, indicating the lower limit of the query prediction hit rate;
[0060] (2) Taking the time window as the unit, the data shard files locked by the previous time window in the current time window are traversed. If the data shard file is not in the data shard file list corresponding to the current time window, the database system interface madvise is called to unlock the locked data shard file.
[0061] The system of this embodiment can perform workload prediction, memory-mapped data preloading, intelligent buffer pool management, dynamic adaptation and feedback optimization, and optimize database data loading and cache management, greatly improving database query efficiency and reducing system I / O latency.
[0062] Example 2:
[0063] The present invention provides a distributed database buffer pool management method, which manages the distributed management system buffer pool through the system disclosed in Example 1, including workload prediction, data preloading, intelligent buffer pool management, and dynamic adaptation and feedback optimization.
[0064] Step S100 Workload prediction: Collect and store the query logs of each database node of the database system, use the query logs as input, and use the trained prediction model to predict the query load arrival of each user in the next time window. Based on the prediction results and the metadata of the database system, generate the data shard file list required for the next time window, group the data shard file list according to the node where the data shard file is located, and send each group of data shard file lists to the data preloading module of the corresponding database node.
[0065] In this application, the prediction model is a prediction model constructed based on a statistical time series analysis method or a deep learning method, which is used to take the query log as input, analyze the query load arrival of each user in each time window based on the query log, and predict the query load arrival of each user in the next time window.
[0066] As a specific implementation of workload prediction, this step includes the following operations:
[0067] (1) Collect and store query logs of each database node in the database system;
[0068] (2) Using the query log as input, the trained prediction model is used to predict the query load arrival of each user in the next time window, and the query load arrival of each user in the next time window is obtained as the prediction result output;
[0069] (3) Based on the prediction results and the metadata of the database system, a list of data shard files corresponding to the hot database table to be queried in the next time window is determined, wherein the list information of the data shard file list includes the node information of the database nodes where all copies of the data shard files are located and the path of the corresponding data shard files;
[0070] (4) The data shard file list is taken as output, the data shard file list is grouped according to the node information of the database node where the data shard file is located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node.
[0071] Step S200: Data preloading: According to the received data shard file list, the data shard file path is obtained through the data preloading module, and the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism.
[0072] As a specific implementation of data preloading, this step includes the following operations:
[0073] (1) Obtain the path of the required data segment file according to the received data segment file list;
[0074] (2) Call the database system's interface madvise and load the data required in the next time window from the virtual memory to the physical memory based on the memory mapping mechanism to reduce the data pulling caused by page fault interruptions during query.
[0075] In this embodiment, when the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism, the data preloading module loads the data in parallel through multi-threading to improve the data pulling efficiency.
[0076] Step S300: Intelligent buffer pool management: Based on the LRU cache elimination strategy of the operating system and according to the memory usage, the data required for the next time window is locked, and the size of the time window and the parameters of the prediction model are adaptively adjusted.
[0077] In this embodiment, the intelligent buffer pool management step includes the following operations:
[0078] L100. Regularly check the memory usage of the database process of the current node. If the usage is less than x% of the maximum available value, execute step L200; otherwise, execute step L300. Here, x% is a database system parameter indicating the database node-level memory usage limit, which can be manually modified by the user.
[0079] L200, call the database system interface madvise to lock the data required in the next time window to prevent the data from being swapped out and improve the memory hit rate;
[0080] L300. Sort the data required in the next time window from high to low priority according to the time dimension, call the database system interface mlock to lock the high-priority data, eliminate the low-priority data, and ensure that the memory usage does not reach the maximum available value. At the same time, reduce the size of the time window and adjust the parameters of the prediction model to ensure that the frequent swapping in and out of cached data due to excessive cached data will occur as little as possible in future time windows.
[0081] Step S400: Dynamic Adaptation and Feedback Optimization: Optimize the parameters of the prediction model based on the query hit rate in the current time window in units of time windows, and unlock the data shard files based on the required data in the current time window.
[0082] As a specific implementation of dynamic adaptation and feedback optimization, this step includes the following operations:
[0083] (1) Taking the time window as the unit, the query hit rate is calculated based on the prediction results of the current time window and the actual results of the previous time window. If the query hit rate is lower than y%, the prediction model is updated and optimized based on the latest query logs collected, where y% is a database system parameter, indicating the lower limit of the query prediction hit rate;
[0084] (2) Taking the time window as the unit, the data shard files locked by the previous time window in the current time window are traversed. If the data shard file is not in the data shard file list corresponding to the current time window, the database system interface madvise is called to unlock the locked data shard file.
[0085] The method of this embodiment optimizes database data loading and cache management by performing workload prediction, memory-mapped data preloading, intelligent buffer pool management, dynamic adaptation and feedback optimization, thereby greatly improving database query efficiency and reducing system I / O latency.
[0086] The distributed database buffer pool management system and method provided by the present invention have been described in detail above. Specific examples have been used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is intended only to facilitate understanding of the method and core concept of the present invention. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.
Claims
1. A distributed database buffer pool management system, characterized in that: It includes workload prediction module, data preloading module, intelligent buffer pool management module and dynamic adaptation and feedback optimization module; The workload prediction module is configured to perform the following operations: collect and store query logs from each database node of the database system, use the query logs as input, and use a trained prediction model to predict the query load arrival of each user in the next time window; generate a list of data shard files required for the next time window based on the prediction results and metadata of the database system; group the list of data shard files according to the nodes where the data shard files are located; and send each group of data shard file lists to a data preloading module of the corresponding database node; The data preloading module is used to map all the data of the database node to the virtual memory, obtain the data shard file path according to the received data shard file list, and load the data required for the next time window from the virtual memory to the physical memory based on the memory mapping mechanism; The intelligent buffer pool management module is used to lock the data required for the next time window based on the LRU cache elimination strategy of the operating system and the memory usage, and adaptively adjust the size of the time window and the parameters of the prediction model; The dynamic adaptation and feedback optimization module is used to optimize the parameters of the prediction model based on the query hit rate in the current time window in units of time windows, and unlock the data shard files based on the required data in the current time window.
2. The distributed database buffer pool management system according to claim 1, characterized in that: The prediction model is a prediction model constructed based on a statistical time series analysis method or a deep learning method, which is used to take query logs as input, analyze the query load arrival of each user in each time window based on the query logs, and predict the query load arrival of each user in the next time window.
3. The distributed database buffer pool management system according to claim 1, characterized in that: The workload prediction module is used to perform the following operations: Collect and store query logs of each database node in the database system; Using query logs as input, the trained prediction model is used to predict the query load arrival of each user in the next time window. The query load arrival of each user in the next time window is obtained as the prediction result output; Based on the prediction results and the metadata of the database system, a list of data shard files corresponding to the hot database table to be queried in the next time window is determined. The list information of the data shard file list includes the node information of the database nodes where all copies of the data shard files are located and the paths of the corresponding data shard files; The data shard file list is used as output, and the data shard file list is grouped according to the node information of the database node where the data shard file is located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node.
4. The distributed database buffer pool management system according to claim 1, characterized in that: The data preloading module is used to perform the following operations: Obtain the path of the required data shard file according to the received data shard file list; The database system's interface madvise is called, and the data required for the next time window is loaded from the virtual memory to the physical memory based on the memory mapping mechanism.
5. The distributed database buffer pool management system according to claim 1, characterized in that: When loading the data required for the next time window from virtual memory to physical memory based on the memory mapping mechanism, the data preloading module loads the data in parallel through multi-threading.
6. The distributed database buffer pool management system according to claim 1, characterized in that: The intelligent buffer pool management module is used to perform the following operations: L100. Regularly check the memory usage of the database process of the current node. If the usage is less than x% of the maximum available value, execute step L200; otherwise, execute step L300. Here, x% is a database system parameter indicating the database node-level memory usage limit, which can be manually modified by the user. L200, call the database system's interface madvise to lock the data required in the next time window; L300, sort the data required in the next time window from high to low priority according to the time dimension from near to far, call the database system's interface mlock to lock the high-priority data and eliminate the low-priority data. At the same time, reduce the size of the time window and adjust the parameters of the prediction model to ensure that in future time windows, there is as little frequent swapping in and out of cached data as possible due to excessive cached data.
7. The distributed database buffer pool management system according to claim 1, characterized in that: The dynamic adaptation and feedback optimization module is used to perform the following operations: The query hit rate is calculated based on the prediction results of the current time window and the actual results of the previous time window, taking the time window as the unit. If the query hit rate is lower than y%, the prediction model is updated and optimized based on the latest query logs collected. y% is a database system parameter that represents the lower limit of the query prediction hit rate. Taking the time window as the unit, the data shard files locked in the previous time window are traversed for the current time window. If the data shard file is not in the data shard file list corresponding to the current time window, the database system interface madvise is called to unlock the locked data shard file.
8. A distributed database buffer pool management method, characterized in that: Managing a distributed management system buffer pool by using a distributed database buffer pool management system according to any one of claims 1 to 7 comprises the following steps: Workload prediction: Query logs from each database node in the database system are collected and stored. Using the query logs as input, the trained prediction model is used to predict the query load arrival of each user in the next time window. Based on the prediction results and the database system's metadata, a list of data shard files required for the next time window is generated. The data shard file lists are grouped according to the nodes where the data shard files are located, and each group of data shard file lists is sent to the data preloading module of the corresponding database node. Data preloading: All data of each database node is loaded into virtual memory. Based on the received data shard file list, the data preloading module obtains the data shard file path and loads the data required for the next time window from virtual memory to physical memory based on the memory mapping mechanism. Intelligent buffer pool management: Based on the operating system's LRU cache elimination strategy, it locks the data required for the next time window according to memory usage, and adaptively adjusts the time window size and prediction model parameters; Dynamic adaptation and feedback optimization: Optimize the prediction model parameters based on the query hit rate in the current time window and unlock the data shard files based on the data required in the current time window.