A database cache management method and system based on deep learning

Abstract

The application provides a kind of database cache management method and system based on deep learning, by predicting and loading the data that may be accessed in the future into cache in advance, effectively reduce the situation of cache miss.In the process of model training, various characteristics of data are fully considered, such as time characteristics, historical access frequency characteristics and the correlation characteristics between tables, so as to improve the accuracy of prediction.Improved long short-term memory network (LSTM) is used, and the association information between database tables is introduced, so that the data access mode is more accurate, and the cache hit rate and system response speed are significantly improved.The application intelligently analyzes and efficiently predicts, optimizes resource utilization, reduces access delay, improves overall system performance, and provides a more intelligent cache management strategy for complex and variable database access patterns.

Description

Technical Field

[0001] This invention relates to a database cache management method and system based on deep learning, belonging to the field of database management. Background Technology

[0002] In modern database systems, caching mechanisms can improve data access efficiency and reduce the number of disk I / O operations. In practical applications, databases typically face high-frequency data access and large-scale data processing demands. To ensure efficient data access, database systems introduce access caching to improve overall access efficiency. These caching mechanisms significantly reduce the time required for direct disk access by storing frequently used data and query results in memory, thereby improving system performance.

[0003] Common cache management methods include buffer pool management, query caching, and distributed caching. One traditional buffer pool management method is based on the First-In-First-Out (FIFO) principle. The basic principle of the FIFO method is that the data that enters the cache first is the first to be evicted. Due to its simple logic and ease of implementation, the FIFO method is widely used in various systems. However, the main drawback of the FIFO method is its low hit rate; it cannot adjust according to data access patterns, which may lead to poor cache performance.

[0004] Another commonly used cache management method is the Least Recently Used (LRU) strategy. The LRU strategy assumes that if data hasn't been accessed recently, it's less likely to be accessed in the future. The LRU strategy performs well in many situations, especially in scenarios with relatively uniform data access patterns. However, for certain specific access patterns, such as repeatedly accessing large datasets, the advantages of the LRU strategy are not significant, and it may even lead to decreased cache efficiency. This is because in this access pattern, data in the cache is frequently replaced, resulting in a significant decrease in the cache hit rate, thereby reducing the overall system performance.

[0005] In conclusion, while traditional cache management methods perform well in certain application scenarios, their limitations are obvious when faced with complex and ever-changing data access patterns. To further improve database system performance, more intelligent and adaptive cache management strategies are needed. Attached Figure Description

[0006] Figure 1 This is a system flowchart. Summary of the Invention

[0007] To address the shortcomings and deficiencies of existing conventional cache management methods, this invention proposes a database cache management method based on deep learning, the specific steps of which are as follows:

[0008] (1) Collect data; extract table access records from the database logs for the past year, including timestamps and table names, organize the access records according to time series, and create an hourly access frequency series;

[0009] (2) Feature extraction: Transform and statistically analyze the timestamps in the database logs to extract features that reflect time dependence and access frequency;

[0010] (3) Define the model: Construct an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit;

[0011] (4) Model training: Forward propagation is performed given the input. When the difference between the predicted value and the true value is too large, backward propagation is performed.

[0012] (5) Data pre-caching calculation: The cache loading process predicts the future access frequency of database tables and loads them into the cache.

[0013] Further, in step (3), the loss function is calculated. Where A jk Let be the association matrix between table j and table k, λ be a hyperparameter controlling the degree of influence of association penalty, N be the sample size, n be the number of sample tables, and y be the number of tables. i For the actual value of the i-th sample, Let be the predicted value for the i-th sample. The squared difference between the predicted values ​​of table j and table k.

[0014] Furthermore, in step (4), during backpropagation, the predicted value is calculated. gradient Use gradient updates.

[0015] Further, in step (5), the calculation is performed. Where sigmoid is the activation function, x t Enter the current time. C is the predicted value at the current moment. t For memory unit output, h t-1 W is the hidden state from the previous moment. y W is the weight matrix of the output layer. i Let b be the weight matrix of the input gate. i Let b be the bias vector of the input gate. y This is the bias vector for the output layer.

[0016] Based on the above method, this invention further proposes a database cache management system based on deep learning, which includes:

[0017] (1) Data collection module: Extract table access records from the database logs for the past year, including timestamps and table names, organize the access records according to time series, and create an hourly access frequency series;

[0018] (2) Feature extraction module: This module transforms and statistically analyzes the timestamps in the database logs to extract features that reflect time dependence and access frequency;

[0019] (3) Model definition module: This module constructs an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit;

[0020] (4) Model training module: Given the input, forward propagation is performed. When the difference between the predicted value and the true value is too large, this module performs back propagation.

[0021] (5) Data pre-caching calculation module: This module loads the cache process, predicts the future access frequency of the database table, and loads it into the cache.

[0022] Furthermore, in the model definition module, the loss function is calculated. Where A jk Let be the association matrix between table j and table k, λ be a hyperparameter controlling the degree of influence of association penalty, N be the sample size, n be the number of sample tables, and y be the number of tables. i For the actual value of the i-th sample, Let be the predicted value for the i-th sample. The squared difference between the predicted values ​​of table j and table k.

[0023] Furthermore, in the model training module, during backpropagation, the predicted values ​​are calculated. gradient Use gradient updates.

[0024] Furthermore, in the data pre-caching calculation module, the calculation... Where sigmoid is the activation function, x t Enter the current time. C is the predicted value at the current moment. t For memory unit output, h t-1 W is the hidden state from the previous moment. y W is the weight matrix of the output layer. i Let b be the weight matrix of the input gate. i Let b be the bias vector of the input gate. y This is the bias vector for the output layer.

[0025] Data pre-caching calculations are based on table access records from the previous 24 hours, currently accessed tables and fields, and current query conditions, which serve as real-time inputs to the model to predict the access frequency of each data table in the next 24 hours. Based on the predicted access frequency, database tables with higher access frequencies are selected and pre-loaded into the read-ahead cache.

[0026] This invention analyzes historical access data in a database, preloads potentially accessed data, and uses deep learning technology to build and train a predictive model to predict data access sequences from database clients. This model can identify patterns and regularities hidden in historical access data, thus more accurately predicting future access behavior. Detailed Implementation

[0027] Depend on Figure 1 As can be seen, based on the prediction model, this invention can preload predicted data into the memory cache before the actual access request arrives, thus optimizing resource utilization efficiency. The main process is as follows:

[0028] (1) Data collection: Extract table access records from the database logs for the past year, including timestamps and table names. Organize the access records according to time series to create an hourly access frequency sequence.

[0029] (2) Feature extraction: By converting and statistically analyzing the timestamps in the database logs, features that reflect time dependence and access frequency are extracted.

[0030] Furthermore, the feature extraction process is as follows:

[0031] The generation of time features decomposes timestamps into fine-grained features such as hours, days of the week, and months to reveal patterns in data access over time.

[0032] Historical access frequency features are calculated based on the access records of each database table within the past time window, determining the average number of accesses over the past 24 hours and the past 7 days.

[0033] Relationship characteristics: Extract the relationship characteristics between database tables, such as tables that are frequently accessed together in the same transaction, or tables that are accessed with similar frequency in similar time periods, and construct a relationship matrix.

[0034] (3) Model Building: Construct an improved LSTM model that includes an input gate, a forget gate, an output gate, and a memory unit. First, define the loss function L and W. i and b i These represent the weights and matrix of the input gate, respectively. Where A is the association matrix between tables, A jkλ represents the association between table j and table k, λ is a hyperparameter controlling the degree of influence of the association penalty, N represents the sample size, n represents the number of sample tables, and y i This represents the actual value of the i-th sample. This represents the predicted value of the i-th sample. This represents the squared difference between the predicted values ​​of table j and table k.

[0035] (4) Model training: The training idea of ​​this model is to perform forward propagation given the input, select a predefined loss function, and perform backpropagation when the difference between the predicted value and the true value is too large to achieve the purpose of parameter update.

[0036] Forward propagation input gate: i t =σ(W i ·[h t-1 ,x t ]+b i ), where h t-1 This indicates the hidden state at the previous moment.

[0037] Backpropagation calculates the gradient: for the predicted value Calculate the gradient as The weight matrix W of the input gate for the loss function L i and bias vector b i The gradient, i.e. and in and These represent the gradients of the predicted value with respect to the weight matrix and the bias vector, respectively.

[0038] Update parameters: Update using the gradient update formula, which is: Where α is the learning rate, W i This represents the weight matrix.

[0039] (5) Data pre-caching calculation module: During actual database access, the cache loading process predicts the future access frequency of database tables and loads the data that may be accessed into the cache. The specific process is as follows: Where, x t This represents the input at the current moment. C represents the predicted value at the current moment. t h represents the output of the memory unit. t-1 W represents the hidden state at the previous moment. y W is the weight matrix of the output layer. i Let b represent the weight matrix of the input gate. i Let b be the bias vector of the input gate. y This represents the bias vector of the output layer.

[0040] This calculation module uses table access records from the previous 24 hours, currently accessed tables and fields, and current query conditions as real-time inputs to predict the access frequency of each data table in the next 24 hours. Based on the predicted access frequency, it selects the database tables with higher access frequencies and loads them into the pre-read cache in advance.

[0041] Technical Effects: This method uses a deep learning model to predict the access frequency of database tables, preloading potentially accessed data into the cache, reducing cache misses, and thus improving cache utilization and overall performance. By analyzing historical access data and extracting time features, historical access frequency features, and correlation features, this method can identify and adapt to complex and ever-changing access patterns, providing a more intelligent cache management strategy. Preloading potentially accessed data allows the database to respond more quickly when actual access requests arrive, reducing data access latency and improving system response speed.

[0042] A more specific example:

[0043] Suppose a database contains three tables: Table1, Table2, and Table3.

[0044]

[0045]

[0046] Access records for these tables were extracted from the database logs over the past year and compiled into an hourly access frequency sequence. Below is a partial data example:

[0047] Time feature generation

[0048] 2023-06-01 08:00:00 8 4 6 2023-06-01 08:05:00 8 4 6 2023-06-01 08:10:00 8 4 6 2023-06-01 08:15:00 8 4 6 … … … …

[0049] Historical access frequency characteristics

[0050] Table 1 10 70 Table 2 8 58 Table 3 5 35

[0051] Relationship characteristics

[0052] Table 1 1 0.5 0.2 Table 2 0.5 1 0.3 Table 3 0.2 0.3 1

[0053] Model Building and Training: Construct an improved LSTM model, including an input gate, forget gate, output gate, and memory unit. Define the loss function. The prediction accuracy is optimized by combining the correlation matrix between the tables.

[0054] Define the parameters and formulas in the LSTM model: Assume the weight matrix and bias vector are initialized as follows:

[0055] b y =0.5

[0056] The training approach of this model is to perform forward propagation given the input, select a predefined loss function, and then perform backpropagation to update the parameters when the difference between the predicted value and the true value is too large.

[0057] Compute the forward propagation input gate i t :i t =σ(W i ·[h t-1 ,x t ]+b i Backpropagation calculates the gradient and updates the weights and biases for the predicted values. Calculate the gradient: The weight matrix W of the input gate for the loss function L i and bias vector b i The gradient, i.e. and Then, the gradient update formula is used for updating, which is: Where α is the learning rate, representing the step size for each update.

[0058] Performing data pre-caching calculations involves loading the cache during actual database accesses, predicting the future access frequency of database tables, and loading the potentially accessed data into the cache. The specific process is as follows: Assume the following conditions: h t-1 =0, C t-1 =0, using the above features and weight matrix for prediction, the following specific predicted values ​​are obtained: These forecasts indicate that Table1, Table2, and Table3 will be accessed 15, 10, and 8 times respectively within the next 24 hours.

[0059] Based on the predicted access frequency, data from Table1 and Table2, which are accessed more frequently, are preloaded into the cache.

[0060] The specific calculation results demonstrate that this deep learning-based database cache management method exhibits significant intelligence and efficiency. By predicting and loading potentially accessed data into the cache in advance, it effectively reduces cache misses. The model training process fully considers various data features, such as temporal characteristics, historical access frequency, and inter-table relationships, thereby improving prediction accuracy. This method employs an improved Long Short-Term Memory (LSTM) network, incorporating inter-table relationships. This makes the method more accurate than traditional methods in predicting data access patterns, significantly improving cache hit rate and system response speed.

[0061] In summary, deep learning-based database cache management methods optimize resource utilization, reduce access latency, and improve overall system performance through intelligent analysis and efficient prediction, providing a more intelligent cache management strategy for complex and ever-changing database access patterns.

[0062] The units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between the devices or units can be electrical, mechanical, or other forms.

[0063] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0064] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0065] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0066] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0067] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A database cache management method based on deep learning, characterized in that: The method includes the following steps: (1) Data collection: Extract table access records from the database logs for the past year, including timestamps and table names, organize the access records according to time series, and create an hourly access frequency series; (2) Feature extraction: Transform and statistically analyze the timestamps in the database logs to extract features that reflect time dependence and access frequency; extract the correlation features between database tables and construct a correlation matrix; (3) Define the model: Construct an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit; (4) Model training: Forward propagation is performed given the input. When the difference between the predicted value and the true value is too large, backward propagation is performed. (5) Data pre-caching calculation: The cache loading process predicts the future access frequency of database tables and loads them into the cache; In step (3), the loss function is calculated. ,in The relationship matrix between table j and table k is shown. To control the impact of correlation penalty, a hyperparameter is defined as N, where N is the number of samples and n is the size of the sample table. Let i be the actual value of the i-th sample. Let be the predicted value for the i-th sample. The squared difference between the predicted values ​​of table j and table k.

2. The database cache management method based on deep learning as described in claim 1, characterized in that: In step (4), during backpropagation, the predicted value is calculated. gradient Gradient updates are used.

3. The database cache management method based on deep learning as described in claim 1, characterized in that: In step (5), calculate ,in, For activation function, Enter the current time. The predicted value at the current moment. For memory unit output, This is the hidden state from the previous moment. The weight matrix of the output layer. Here is the weight matrix of the input gate. This is the bias vector for the input gate. This is the bias vector for the output layer.

4. A database cache management system based on deep learning, characterized in that: The system includes: (1) Data collection module: Extract table access records from the database logs for the past year, including timestamps and table names, organize the access records according to time series, and create an hourly access frequency sequence; (2) Feature extraction module: This module transforms and statistically analyzes the timestamps in the database logs, extracts features that reflect time dependence and access frequency; extracts the correlation features between database tables and constructs a correlation matrix; (3) Model definition module: This module constructs an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit; (4) Model training module: Given the input, forward propagation is performed. When the difference between the predicted value and the true value is too large, this module performs back propagation. (5) Data pre-caching calculation module: This module loads the cache process, predicts the future access frequency of the database table, and loads it into the cache; In the model definition module, the loss function is calculated. ,in The relationship matrix between table j and table k is shown. To control the impact of correlation penalty, a hyperparameter is defined as N, where N is the number of samples and n is the size of the sample table. Let i be the actual value of the i-th sample. Let be the predicted value for the i-th sample. The squared difference between the predicted values ​​of table j and table k.

5. A database cache management system based on deep learning as described in claim 4, characterized in that: In the model training module, during backpropagation, the predicted values ​​are calculated. gradient Gradient updates are used.

6. A database cache management system based on deep learning as described in claim 4, characterized in that: In the data pre-caching calculation module, the calculation ,in, For activation function, Enter the current time. The predicted value at the current moment. For memory unit output, This is the hidden state from the previous moment. The weight matrix of the output layer. Here is the weight matrix of the input gate. This is the bias vector for the input gate. This is the bias vector for the output layer.

Images