Database cache management method and system based on deep learning

Abstract

According to the database cache management method and system based on deep learning provided by the invention, the situation of cache miss is effectively reduced by predicting and loading data which is possibly accessed in the future into the cache in advance. In the model training process, various characteristics of data are fully considered, such as time characteristics, historical access frequency characteristics and relevance characteristics between tables, so that the prediction accuracy is improved. An improved long short-term memory (LSTM) network is adopted, and associated information between database tables is introduced, so that the prediction of a data access mode is more accurate, and the cache hit rate and the system response speed are remarkably improved. Through intelligent analysis and efficient prediction, resource utilization is optimized, access delay is reduced, the overall system performance is improved, and a more intelligent cache management strategy is provided for a complex and changeable database access mode.

Description

Technical Field

[0001] The present invention relates to a database cache management method and system based on deep learning, and belongs to the field of database management. Background Art

[0002] In modern database systems, cache mechanisms can improve data access efficiency and reduce the number of disk I / O operations. In actual applications, databases are usually faced with high-frequency data access and large amounts of data processing requirements. In order to ensure efficient data access, database systems introduce access caches to improve overall access efficiency. These cache mechanisms greatly reduce the time required to directly access the disk by storing commonly used data and query results in memory, thereby improving system performance.

[0003] Common cache management methods include buffer pool management, query cache, and distributed cache. One of the traditional buffer pool management methods is a cache management method based on the first-in-first-out (FIFO) principle. The basic principle of the FIFO method is that the data that enters the cache first is eliminated first. Due to its simple logic and easy implementation, the FIFO method is widely used in various systems. However, the main disadvantage of the FIFO method is its low hit rate. It cannot be adjusted according to the access pattern of the data, which may lead to poor cache performance.

[0004] Another commonly used cache management method is the Least Recently Used (LRU) strategy. The LRU strategy assumes that if some data has not been accessed in the recent period, it is less likely to be accessed in the future. The LRU strategy performs well in many cases, especially in scenarios where the data access pattern is relatively uniform. However, for certain access patterns, such as looping access to large data sets, the advantages of the LRU strategy are not obvious and may even lead to a decrease in cache efficiency. This is because under this access pattern, the data in the cache is frequently replaced, resulting in a significant decrease in the cache hit rate, thereby reducing the overall performance of the system.

[0005] In summary, although the traditional cache management method performs well in some application scenarios, its limitations are obvious when facing complex and changing data access patterns. In order to further improve the performance of the database system, it is necessary to introduce a more intelligent and adaptive cache management strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is the system flow chart. Summary of the invention

[0007] In view of the defects and shortcomings of the existing conventional cache management methods, the present invention proposes a database cache management method based on deep learning, the specific steps are:

[0008] (1) Collect data; extract table access records from the database log for the past year, including timestamps and table names, organize the access records in time series, and create an hourly access frequency sequence;

[0009] (2) Feature extraction: Convert and count the timestamps in the database logs to extract features that reflect time dependency and access frequency;

[0010] (3) Define the model: Build an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit;

[0011] (4) Model training: forward propagation is performed on a given input. When the difference between the predicted value and the true value is too large, back propagation is performed.

[0012] (5) Data pre-cache calculation: Load cache process, predict the future access frequency of the database table, and load it into the cache.

[0013] Furthermore, in step (3), the loss function is calculated Among them A jk is the correlation matrix between table j and table k, λ is the hyperparameter that controls the degree of correlation penalty, N is the number of samples, n is the number of sample tables, y i is the actual value of the i-th sample, is the predicted value of the i-th sample, is the squared difference between the predicted values ​​in table j and table k.

[0014] Furthermore, in step (4), during back propagation, the predicted value is calculated gradient Utilize gradient updates.

[0015] Furthermore, in step (5), calculate Among them, sigmoid is the activation function, x t Input for the current moment, is the current moment prediction value, C t is the memory unit output, h t-1 is the hidden state at the previous moment, W y is the weight matrix of the output layer, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, b y is the bias vector of the output layer.

[0016] Based on the above method, the present invention further proposes a database cache management system based on deep learning, which includes:

[0017] (1) Data collection module: extract table access records from the database log for the past year, including timestamps and table names, organize the access records according to time series, and create an hourly access frequency sequence;

[0018] (2) Feature extraction module: This module converts and counts the timestamps in the database logs and extracts features that reflect time dependency and access frequency;

[0019] (3) Model definition module: This module constructs an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit;

[0020] (4) Model training module: forward propagation is performed on a given input. When the difference between the predicted value and the true value is too large, the module performs back propagation.

[0021] (5) Data pre-cache calculation module: This module loads the cache process, predicts the future access frequency of the database table, and loads it into the cache.

[0022] Furthermore, in the model definition module, the loss function is calculated Among them A jk is the correlation matrix between table j and table k, λ is the hyperparameter that controls the degree of correlation penalty, N is the number of samples, n is the number of sample tables, y i is the actual value of the i-th sample, is the predicted value of the i-th sample, is the squared difference between the predicted values ​​in table j and table k.

[0023] Furthermore, in the model training module, during back propagation, the predicted value is calculated gradient Utilize gradient updates.

[0024] Furthermore, in the data pre-caching calculation module, the calculation Among them, sigmoid is the activation function, x t Input for the current moment, is the current moment prediction value, C t is the memory unit output, h t-1 is the hidden state at the previous moment, W y is the weight matrix of the output layer, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, b y is the bias vector of the output layer.

[0025] Data pre-cache calculation is based on the table access records in the previous 24 hours, the currently accessed table and field, and the current query conditions as real-time input to the model to predict the access frequency of each data table in the next 24 hours. Based on the predicted access frequency, database tables with higher access frequencies are selected and loaded into the pre-read cache in advance.

[0026] The present invention analyzes the historical access data in the database, preloads the data that may be accessed, and uses deep learning technology to build and train a prediction model to predict the data access sequence of the database client. The model can identify the patterns and regularities hidden in the historical access data, so as to more accurately predict future access behavior. DETAILED DESCRIPTION

[0027] Depend on Figure 1 It can be seen that based on the prediction model, the present invention can load the predicted data into the memory cache in advance before the actual access request arrives, thereby optimizing resource utilization efficiency. The main process is:

[0028] (1) Data collection: Extract table access records from the database logs for the past year, including timestamps and table names. Arrange the access records in time series and create an hourly access frequency sequence.

[0029] (2) Feature extraction: By converting and counting the timestamps in the database log, features that can reflect time dependency and access frequency can be extracted.

[0030] Furthermore, the feature extraction process is as follows:

[0031] The generation of temporal features decomposes the timestamp into fine-grained features such as hour, week, and month to reveal the temporal patterns of data access.

[0032] The historical access frequency feature calculates the average number of accesses in the past 24 hours and the past 7 days based on the access records of each database table in the past time window.

[0033] Correlation features: Extract correlation features between database tables, such as tables that are often accessed together in the same transaction, or tables with similar access frequencies in a similar time period, and build a correlation matrix.

[0034] (3) Model building: Build an improved LSTM model that includes an input gate, a forget gate, an output gate, and a memory unit. First, define the loss function L, W i and b i Represent the weight and matrix of the input gate respectively, Where A is the correlation matrix between tables, A jkrepresents the correlation between table j and table k, λ is a hyperparameter that controls the degree of correlation penalty, N represents the number of samples, n represents the number of sample tables, and y i represents the actual value of the i-th sample, represents the predicted value of the i-th sample, represents the squared difference between the predicted values ​​of table j and table k.

[0035] (4) Model training: The training idea of ​​this model is to perform forward propagation for a given input and select a defined loss function. When the gap between the predicted value and the true value is too large, back propagation will be performed to achieve the purpose of parameter update.

[0036] Forward propagation input gate: i t =σ(W i ·[h t-1 ,x t ]+b i ), where h t-1 Represents the hidden state at the previous moment.

[0037] Back propagation calculates the gradient: the predicted value The gradient is calculated as The loss function L has a weight matrix W for the input gate i and the bias vector b i The gradient of and in and Represent the gradient of the predicted value with respect to the weight matrix and bias vector respectively.

[0038] Update parameters: Update using the gradient update formula, the formula is: Where α is the learning rate, W i represents the weight matrix.

[0039] (5) Data pre-cache calculation module: In actual database access, the cache process is loaded to predict the future access frequency of the database table and load the data that may be accessed into the cache. The specific process is as follows: Among them, x t Represents the input at the current moment, represents the predicted value at the current moment, C t Represents the memory unit output, h t-1 represents the hidden state at the previous moment, W y is the weight matrix of the output layer, W i represents the weight matrix of the input gate, b i is the bias vector of the input gate, b y Represents the bias vector of the output layer.

[0040] This calculation module uses the table access records in the previous 24 hours, the currently accessed table and field, and the current query conditions as real-time inputs to the model to predict the access frequency of each data table in the next 24 hours. Based on the predicted access frequency, the database table with a higher access frequency is selected and loaded into the pre-read cache in advance.

[0041] Technical effect: This method predicts the access frequency of database tables through deep learning models, loads data that may be accessed in the future into the cache in advance, reduces cache misses, and thus improves cache utilization and overall performance. This method analyzes historical access data, extracts time features, historical access frequency features, and correlation features, and can identify and adapt to complex and changing access patterns, providing a smarter cache management strategy. Preloading data that may be accessed enables the database to respond more quickly when actual access requests arrive, reduces data access delays, and improves system response speed.

[0042] A more specific embodiment:

[0043] Assume that a database contains three tables: Table1, Table2, and Table3.

[0044]

[0045]

[0046] In the database logs of the past year, we extracted the access records of these tables and sorted them into hourly access frequency sequences. The following is a sample of some of the data:

[0047] Time feature generation

[0048] Timestamp Hour Day of the week month 2023-06-01 08:00:00 8 4 6 2023-06-01 08:05:00 8 4 6 2023-06-01 08:10:00 8 4 6 2023-06-01 08:15:00 8 4 6 … … … …

[0049] Historical access frequency characteristics

[0050] Table name Number of visits in the past 24 hours Number of visits in the past 7 days Table 1 10 70 Table 2 8 58 Table 3 5 35

[0051] Relevance Features

[0052] Table name Table 1 Table 2 Table 3 Table 1 1 0.5 0.2 Table 2 0.5 1 0.3 Table 3 0.2 0.3 1

[0053] Model building and training: Build an improved LSTM model, including input gate, forget gate, output gate and memory unit. Define the loss function Combine the correlation matrix between tables to optimize prediction accuracy.

[0054] Define the parameters and formulas in the LSTM model: Assume that the weight matrix and bias vector are initialized as:

[0055] b y =0.5

[0056] The training idea of ​​this model is to perform forward propagation for a given input and select a defined loss function. When the gap between the predicted value and the true value is too large, backpropagation will be performed to achieve the purpose of parameter update.

[0057] Calculate the forward propagation input gate i t :i t =σ(W i ·[h t-1 ,x t ]+b i ), back propagation calculates the gradient and updates the weights and biases, and the predicted value Compute the gradient: The loss function L has a weight matrix W for the input gate i and the bias vector b i The gradient of and Then use the gradient update formula to update, the formula is: Where α is the learning rate, which represents the step size of each update.

[0058] Perform data pre-cache calculation, that is, in actual database access, load the cache process, predict the future access frequency of the database table, and load the data that may be accessed into the cache. The specific process is: Assume the following situation: h t-1 =0, C t-1 =0, using the above features and weight matrix for prediction, the following specific prediction values ​​are obtained: These predicted values ​​indicate that in the next 24 hours, the access frequencies of Table1, Table2, and Table3 will be 15, 10, and 8, respectively.

[0059] According to the predicted access frequency, the data of the tables Table1 and Table2 with higher access frequency are loaded into the cache in advance.

[0060] Judging from the specific calculation results, the database cache management method based on deep learning demonstrates significant intelligence and efficiency. By predicting and loading data that may be accessed in the future into the cache in advance, cache misses are effectively reduced. During the model training process, multiple data features, such as time features, historical access frequency features, and correlation features between tables, are fully considered to improve the accuracy of prediction. This method uses an improved long short-term memory network (LSTM) and introduces correlation information between database tables. This makes this method more accurate than traditional methods in predicting data access patterns, significantly improving cache hit rates and system response speed.

[0061] In general, the database cache management method based on deep learning optimizes resource utilization, reduces access latency, and improves overall system performance through intelligent analysis and efficient prediction, providing a smarter cache management strategy for complex and changeable database access patterns.

[0062] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0063] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0064] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0065] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0066] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0067] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A database cache management method based on deep learning, characterized in that: The method comprises the following steps: (1) Data collection: Extract table access records from the database log for the past year, including timestamps and table names, organize the access records in time series, and create an hourly access frequency sequence; (2) Feature extraction: Convert and count the timestamps in the database logs to extract features that reflect time dependency and access frequency; (3) Define the model: Build an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit; (4) Model training: forward propagation is performed on a given input. When the difference between the predicted value and the true value is too large, back propagation is performed. (5) Data pre-cache calculation: Load cache process, predict the future access frequency of the database table, and load it into the cache.

2. A database cache management method based on deep learning as claimed in claim 1, characterized in that: In step (3), the loss function is calculated Among them A jk is the correlation matrix between table j and table k, λ is the hyperparameter that controls the degree of correlation penalty, N is the number of samples, n is the number of sample tables, y i is the actual value of the i-th sample, is the predicted value of the i-th sample, is the squared difference between the predicted values ​​in table j and table k.

3. The database cache management method based on deep learning according to claim 1, characterized in that: In step (4), during back propagation, the predicted value is calculated gradient Utilize gradient updates.

4. The database cache management method based on deep learning according to claim 1, characterized in that: In step (5), calculate Among them, sigmoid is the activation function, x t Input for the current moment, is the current moment prediction value, C t is the memory unit output, h t-1 is the hidden state at the previous moment, W y is the weight matrix of the output layer, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, b y is the bias vector of the output layer.

5. A database cache management system based on deep learning, characterized in that: The system includes: (1) Data collection module: extracts table access records from the database logs for the past year, including timestamps and table names, organizes the access records in time series, and creates an hourly access frequency sequence; (2) Feature extraction module: This module converts and counts the timestamps in the database logs and extracts features that reflect time dependency and access frequency; (3) Model definition module: This module constructs an improved LSTM model, which includes an input gate, a forget gate, an output gate, and a memory unit; (4) Model training module: forward propagation is performed on a given input. When the difference between the predicted value and the true value is too large, the module performs back propagation. (5) Data pre-cache calculation module: This module loads the cache process, predicts the future access frequency of the database table, and loads it into the cache.

6. A database cache management system based on deep learning as claimed in claim 5, characterized in that: In the model definition module, calculate the loss function Among them A jk is the correlation matrix between table j and table k, λ is the hyperparameter that controls the degree of correlation penalty, N is the number of samples, n is the number of sample tables, y i is the actual value of the i-th sample, is the predicted value of the i-th sample, is the squared difference between the predicted values ​​in table j and table k.

7. A database cache management system based on deep learning as claimed in claim 5, characterized in that: In the model training module, during back propagation, the predicted value is calculated gradient Utilize gradient updates.

8. A database cache management system based on deep learning as claimed in claim 5, characterized in that: In the data pre-cache calculation module, calculation Among them, sigmoid is the activation function, x t Input for the current moment, is the current moment prediction value, C t is the memory unit output, h t-1 is the hidden state at the previous moment, W y is the weight matrix of the output layer, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, b y is the bias vector of the output layer.

Images