Database cache optimization method and device, equipment, medium and product

Abstract

The invention relates to a database cache optimization method and device, equipment, a medium and a product. The method comprises the following steps: performing state representation on current cache state parameters cached in a database to obtain current state representation data, and determining a current cache optimization space according to the current state representation data; determining a current cache optimization action according to a reinforcement learning model, the current state representation data and the current cache optimization space; and optimizing the database cache according to the current cache optimization action. The method can better adapt to dynamically changing database requirements, and the overall performance of the database is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database management, and in particular to a database cache optimization method, device, equipment, medium and product. BACKGROUND

[0002] The database cache is used for temporarily storing frequently accessed data in the database, such as query results, index data, table data, etc., which can reduce the number of accesses to the disk, thereby speeding up the reading and writing of data and improving the access performance and response speed of the database.

[0003] In the conventional technology, the access and scheduling of data in the database cache are managed by a cache optimizer. However, the conventional cache optimizer is difficult to cope with dynamically changing database requirements, thereby affecting the overall performance of the database. SUMMARY

[0004] Therefore, it is necessary to provide a database cache optimization method, device, equipment, medium and product which can better adapt to dynamically changing database requirements and improve the overall performance of the database.

[0005] In a first aspect, the present application provides a database cache optimization method, comprising:

[0006] performing state representation on current cache state parameters of the database cache to obtain current state representation data, and determining a current cache optimization space according to the current state representation data;

[0007] determining a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space;

[0008] optimizing the database cache according to the current cache optimization action.

[0009] In one embodiment, determining the current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space comprises:

[0010] extracting a current state vector from the current state representation data;

[0011] inputting the current state vector and space information of the current cache optimization space into the reinforcement learning model, and determining the current cache optimization action in the current cache optimization space by a main policy network in the reinforcement learning model.

[0012] In one embodiment, the above method further comprises:

[0013] obtaining target cache state parameters corresponding to the optimized data cache as next cache state parameters;

[0014] The target cache state parameter is represented to obtain target state representation data, and the target cache optimization space is determined according to the target state representation data;

[0015] According to the reinforcement learning model, the target state representation data and the target cache optimization space, a target cache optimization action is determined;

[0016] According to the target cache optimization action, the optimized database cache is optimized again.

[0017] In one of the embodiments, the method further comprises:

[0018] Obtaining database performance feedback data corresponding to the optimized data cache;

[0019] According to the database performance feedback data, the performance of the database corresponding to the optimized database cache is evaluated.

[0020] In one of the embodiments, the method further comprises:

[0021] Based on the training set, a first loss function value of a main policy network in the deep neural network is determined, and the network parameters of the main policy network are updated according to the first loss function value;

[0022] Based on the training set, a target policy network and a target value network in the deep neural network, a second loss function value of a main value network in the deep neural network is determined, and the network parameters of the main value network are updated according to the second loss function value;

[0023] According to the updated network parameters of the main policy network, the network parameters of the target policy network in the deep neural network are updated, and according to the updated network parameters of the main value network, the network parameters of the target value network in the deep neural network are updated, the updated target policy network and the updated target value network are respectively taken as the target policy network and the target value network, the step of determining the first loss function value of the main policy network in the deep neural network based on the training set is returned, and the model training is continued until the training stop condition is met, and the current main policy network is taken as the reinforcement learning model.

[0024] In one of the embodiments, the method further comprises:

[0025] For each experience sample in the pre-created experience replay buffer, a first value, a second value and a reward value corresponding to the experience sample are obtained, and a time difference error of the experience sample is determined according to the first value, the second value and the reward value corresponding to the experience sample; wherein the first value is output by the main value network, and the second value is output by the target value network;

[0026] The sample sampling probability is determined according to the time difference error;

[0027] The training set is selected from the experience replay buffer according to a sample sampling probability.

[0028] In one of the embodiments, determining the first loss function value of the main policy network in the deep neural network based on the training set comprises:

[0029] The training set is input into the deep neural network, and each experience sample in the training set is extracted by the main policy network of the deep neural network to obtain a cache optimization strategy corresponding to each experience sample;

[0030] The first value of the cache optimization strategy corresponding to each experience sample is determined by the main value network of the deep neural network;

[0031] The first loss function value is determined according to the first value of the cache optimization strategy corresponding to each experience sample.

[0032] In one of the embodiments, determining the second loss function value of the main value network in the deep neural network based on the training set comprises:

[0033] For each experience sample in the training set, the next experience sample of the experience sample is extracted by the target policy network of the deep neural network to obtain a cache optimization strategy of the next experience sample, and the first value of the cache optimization strategy of the next experience sample is determined by the target value network of the deep neural network;

[0034] The reward value corresponding to the experience sample is obtained, and the target value is determined according to the reward value and the first value of the cache optimization strategy of the next experience sample;

[0035] The second loss function value is determined according to the first value of the cache optimization strategy of each experience sample and the target value.

[0036] In a second aspect, the present application also provides a database cache optimization device, comprising:

[0037] A definition module is configured to perform state representation on current cache state parameters of the database cache to obtain current state representation data, and determine a current cache optimization space according to the current state representation data.

[0038] A prediction module is configured to determine a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space.

[0039] An optimization module is configured to optimize the database cache according to the current cache optimization action.

[0040] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0041] state representation data, and determining a current cache optimization space according to the current state representation data;

[0042] determining a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space;

[0043] optimizing the database cache according to the current cache optimization action.

[0044] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0045] state representation data, and determining a current cache optimization space according to the current state representation data;

[0046] determining a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space;

[0047] optimizing the database cache according to the current cache optimization action.

[0048] In a fifth aspect, the present application also provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the following steps:

[0049] state representation data, and determining a current cache optimization space according to the current state representation data;

[0050] determining a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space;

[0051] optimizing the database cache according to the current cache optimization action.

[0052] The above database cache optimization method, device, equipment, medium and product can output a current cache optimization action by introducing a reinforcement learning model, can combine the characteristics of real-time cache state parameters of the database cache, can optimize the database cache in real time, can adapt to a constantly changing database cache environment, and thus improves the overall performance and I / O efficiency of the database. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0054] Figure 1 A flowchart of a database cache optimization method in an embodiment;

[0055] Figure 2 A flowchart of determining a current cache optimization action according to a reinforcement learning model, current state representation data and a current cache optimization space in an embodiment;

[0056] Figure 3 A training diagram of a reinforcement learning model in a database cache optimizer in an embodiment;

[0057] Figure 4 A query execution process diagram of SQL in a reinforcement learning scenario in an embodiment;

[0058] Figure 5 A structural block diagram of a database cache optimization apparatus in an embodiment;

[0059] Figure 6 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0061] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion.

[0062] In an embodiment, as shown in Figure 1 A database cache optimization method is provided, and the present embodiment is exemplified by the method applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. In the present embodiment, the method includes the following steps:

[0063] Step 102, state representation is performed on the current cache state parameters of the database cache to obtain current state representation data, and a current cache optimization space is determined according to the current state representation data.

[0064] The current cache state parameters refer to parameters related to the data cache state, which can include a query queue, cache state information, a cache hit rate during query queue execution, and the like. The current state representation data refers to state data that can be directly input to the reinforcement learning model, that is, state data that can be understood by the reinforcement learning model. The current cache optimization space refers to the optimization action range of the database cache.

[0065] In some embodiments, for a relational database that is running stably for a long time, the current cache data parameters of the database cache of the database are obtained by the database cache optimizer running in the terminal, and state representation and action space definition are performed on the current cache data parameters: the current cache data parameters of the database are defined as S, that is, the current state representation data, S is represented by a bitmap, wherein a row represents a basic relation, and a column represents a data block. If the jth data block of the basic relation i is cached in the buffer, the (i, j) entry is set to 1, otherwise it is set to 0. At the same time, each query q in the query queue points to the data block accessed by the query q in the cache. Based on S, the current cache optimization space can include continuous actions: allocating how much cache proportion to a certain relation, preloading a certain time interval of data volume, adjusting the priority of certain data blocks, etc., and can also include discrete actions: retaining the page / evicting the page, preloading a certain block to adjust the replacement strategy, marking a certain block as hot data, etc.

[0066] The current state representation data is used to depict the current running state of the database cache, so that the reinforcement learning model, that is, the agent, can perceive the system environment. The action space definition is used to limit the cache tuning operations executable by the agent, so that the agent can make optimal decisions in a discrete or continuous action set. Clear modeling of states and actions is the basis for building a reinforcement learning cache optimizer, and determines the trainability and policy effectiveness of the model.

[0067] Further, after obtaining the current cache data parameters of the database cache of the database, data cleaning is performed on the current cache data parameters, including data deduplication to reduce unnecessary interference on decision making, missing value processing to ensure data integrity, and outlier detection method for abnormal value processing.

[0068] Step 104, a current cache optimization action is determined according to the reinforcement learning model, the current state representation data, and the current cache optimization space.

[0069] Step 106, the database cache is optimized according to the current cache optimization action.

[0070] In some implementations, the reinforcement learning model includes a master policy network. The master policy network, current state representation data, and the current cache optimization space determine the current cache optimization action. For example, the current cache optimization action may include: performing preloading, retention, or eviction operations on target data blocks or pages; adjusting the target cache ratio and retention priority of data from different business tables, partitions, or time intervals in the buffer pool; dynamically adjusting the cache replacement strategy and its parameters, including the hot / cold data partitioning threshold and replacement weight coefficient; and controlling the preloading depth, batch size, and whether asynchronous prefetching is enabled. Subsequently, the database cache is optimized based on the current cache optimization action to obtain the optimized database cache.

[0071] In the above-mentioned database cache optimization method, by introducing a reinforcement learning model, the current cache optimization action can be output based on the characteristics of the real-time cache state parameters of the database cache, and the database cache can be optimized in real time. This can adapt to the constantly changing database cache environment, thereby improving the overall performance and I / O efficiency of the database.

[0072] In one exemplary embodiment, such as Figure 2 As shown, based on the reinforcement learning model, the current state representation data, and the current cache optimization space, the current cache optimization actions are determined as follows:

[0073] Step 204: Extract the current state vector from the current state representation data.

[0074] Step 206: Input the current state vector and the spatial information of the current cache optimization space into the reinforcement learning model, and the main policy network in the reinforcement learning model determines the current cache optimization action in the current cache optimization space.

[0075] In some implementations, since the current state representation data is bitmap data, it is necessary to extract the current state vector s from the bitmap data, and then input the current state vector s and the spatial information of the current cache optimization space into the reinforcement learning model. The spatial information of the current cache optimization space refers to the continuous and discrete actions contained within it. The reinforcement learning model is the master policy network, which includes a multilayer perceptron structure. The multilayer perceptron structure progressively extracts the state features of the current state vector s, ultimately outputting a specific optimization action as the current cache optimization action. .

[0076] In the embodiments of the present application, by jointly inputting the current state vector and the space information of the current cache optimization space into the reinforcement learning model, the main policy network can accurately match the real-time state of the system and make optimal cache tuning decisions within the range of executable actions, thereby improving cache hit rate, stability and overall performance of the database system, and ensuring the safety and controllability of the policy selection process. In addition, adaptive cache query optimization can also be achieved: the hit rate, page dirt ratio, LRU aging curve and other states of the cache layer are encoded into low-dimensional vectors and directly input into the reinforcement learning model, so that the reinforcement learning model can "perceive" the cache temperature before each I / O request arrives, and the query scheduling problem is converted into an end-to-end decision of "cache state -> scheduling action" instead of traditional rule-based queuing.

[0077] In an exemplary embodiment, the above method further comprises: obtaining a target cache state parameter corresponding to the optimized data cache as a next cache state parameter; performing state representation on the target cache state parameter to obtain target state representation data, and determining a target cache optimization space according to the target state representation data; determining a target cache optimization action according to the reinforcement learning model, the target state representation data and the target cache optimization space; and re-optimizing the optimized database cache according to the target cache optimization action.

[0078] In some embodiments, the data cache optimization process can be optimized in real time, and after optimizing the database cache according to the current cache optimization action, the optimized database cache can be re-optimized according to the above optimization method.

[0079] In the embodiments of the present application, by optimizing the database cache in real time and re-executing cache tuning according to the optimized cache state after each optimization, the cache can continuously adapt to changes in business load, improve cache hit rate and system throughput, and significantly enhance the dynamic, adaptive and long-term stability of the cache management strategy.

[0080] In an exemplary embodiment, the above method further comprises: obtaining database performance feedback data corresponding to the optimized data cache; and performing performance evaluation on the database corresponding to the optimized database cache according to the database performance feedback data.

[0081] After executing the current cache optimization action, i.e., optimizing the database cache according to the current cache optimization action, database performance feedback data of the database can be obtained. The database performance feedback data can include database throughput, baseline performance, cache hit rate and total block request number. Performance evaluation on the database can be performed according to the database performance feedback data. The performance evaluation can use a reward function, and the cache hit rate of the executed query is used as the reward function, and the specific formula is:

[0082]

[0083] wherein, represents the database throughput, represents the baseline performance, H represents the cache hit rate, represents the total number of block requests, and respectively represent hyperparameters. The database throughput and the cache hit rate are improved to generate positive incentives. The reward function evaluates the effect of each cache optimization action, assesses the improvement or reduction of the database performance through the performance indicators and the change of the cache hit rate, and updates the reinforcement learning model according to the feedback of the reward function.

[0084] In the embodiments of the present application, the database performance feedback data is obtained based on the optimized data cache, and the performance evaluation is performed on the database performance feedback data, so that the cache tuning process forms a verifiable and feedback closed-loop mechanism, the reliability and accuracy of the optimization effect are improved, and the continuous improvement of the database performance and the self-adaptive evolution of the cache management strategy are realized.

[0085] In an exemplary embodiment, the above method further comprises a training step of the reinforcement learning model, including: determining a first loss function value of a main policy network in the deep neural network based on a training set, and updating network parameters of the main policy network according to the first loss function value; determining a second loss function value of a main value network in the deep neural network based on the training set, a target policy network and a target value network in the deep neural network, and updating network parameters of the main value network according to the second loss function value; updating network parameters of the target policy network in the deep neural network according to the updated network parameters of the main policy network, and updating network parameters of the target value network in the deep neural network according to the updated network parameters of the main value network; returning to the step of determining the first loss function value of the main policy network in the deep neural network based on the training set, continuing the model training until a training stop condition is met, and taking the current main policy network as the reinforcement learning model.

[0086] In some embodiments, as Figure 3 shown, a training schematic diagram of the reinforcement learning model in the database cache optimizer.

[0087] A deep neural network is used to construct an intelligent agent for database tuning. The intelligent agent is composed of four core networks: two policy networks (a main policy network Actor1 and a target policy network Actor2) and two value networks (a main value network Critic1 and a target value network Critic2), which respectively undertake different functions and update rhythms.

[0088] First, the policy network is responsible for learning the deterministic policy for database tuning. The main policy network (Actor1) receives the current state s of the environment as input, extracts state features step by step through a multilayer perceptron structure, and finally outputs a specific tuning action. To improve training stability, a target policy network (Actor2) is introduced. This is a delayed copy of the main policy network, whose parameters are softly updated or periodically synchronized at a slower pace during training. This design helps mitigate fluctuations during training, making policy updates smoother.

[0089] The value network is used to evaluate the value of the current state-action pair. The main value network (Critic1) takes the state and action together as input, performs feature fusion and abstraction through a deep fully connected structure, and outputs the corresponding Q-value estimate Q(s, Similarly, a target value network (Critic2) is maintained, whose parameters are updated less frequently than those of the main network, to calculate the target Q-value. This "slow update" mechanism effectively reduces the variance of the target value and enhances the convergence and robustness of the training process.

[0090] In terms of network architecture design, both the policy network and the value network adopt a deep fully connected architecture, but there are some differences in the internal details. The policy network introduces batch normalization and activation functions before the output layer to enhance nonlinear expressiveness and stabilize training; while the value network uses a concatenation and joint encoding method in the state and action fusion stage to ensure that action information is effectively embedded in the state representation. In addition, residual connections or dropout (random deactivation) mechanisms are introduced between all network layers to alleviate the gradient vanishing and overfitting problems in deep networks, and improve the model's performance and generalization performance in complex database environments.

[0091] The reinforcement learning model is trained using historical data, and its decision-making strategy is optimized through interaction with the environment. The database performance is then gradually improved. The specific process is as follows:

[0092] 1) Initialize the network parameters of the deep neural network's Actor network (main policy network Actor1 and target policy network Actor2) and Critic network (main value network Critic1 and target value network Critic2), and create an experience replay buffer (priority replay pool) to store experience samples.

[0093] 2) Interact with the environment and collect sample data. For each time step, select an action using the master policy network based on the current state s. and perform the action, in which This is a noise term. Then observe the next state. and reward value r, and (s, , r, ) as experience samples into the experience replay buffer.

[0094] 3) Experience replay, i.e. the step of selecting a training set from the experience replay buffer. To further improve the training efficiency, a prioritized experience replay mechanism is introduced. This mechanism gives different sampling weights according to the importance of the samples, so that the experiences that contribute more to the training are used more frequently. In this way, the network can focus on key learning signals more quickly and avoid wasting computing resources on invalid or redundant data, thereby speeding up the overall learning process.

[0095] The step of selecting a training set from the experience replay buffer includes: for each experience sample in the pre-created experience replay buffer, obtaining a first value, a second value and a reward value corresponding to the experience sample, and determining a temporal difference error of the experience sample according to the first value, the second value and the reward value corresponding to the experience sample; wherein the first value is output by the main value network, and the second value is output by the target value network; determining a sample sampling probability according to the temporal difference error; and selecting a training set from the experience replay buffer according to the sample sampling probability.

[0096] For each experience sample in the pre-created experience replay buffer, a first value, a second value and a reward value corresponding to the experience sample are obtained, and a temporal difference error (TD error) of the experience sample is determined according to the first value, the second value and the reward value corresponding to the experience sample. A sample sampling probability is determined according to the temporal difference error; and a training set is selected from the experience replay buffer according to the sample sampling probability. The TD error measures the correction amplitude of the algorithm, and the greater the absolute value of the TD error, the greater the correction effect of the sample on the network. The TD error is defined as follows:

[0097]

[0098] wherein, TD error of the experience sample, reward value of the experience sample, which can be calculated according to the reward function, , main value network and target value network, second value output by the target value network, first value output by the main value network.

[0099] Using a probability sampling mechanism can ensure that samples with small TD errors are still sampled, ensuring the diversity of samples during algorithm training. Through experience-based replay training: instead of uniformly sampling past experiences, each trajectory is scored based on "cache stall severity", and the higher the stall, the higher the sample weight; dynamic weighting during playback forces the network to repeatedly learn critical scenarios that have caused cache avalanches, significantly reducing the probability of re-triggering.

[0100] 4) Update the main policy network in the deep neural network. Determine the first loss function value of the main policy network in the deep neural network based on the training set, and update the network parameters of the main policy network according to the first loss function value.

[0101] 5) Update the main value network in the deep neural network. Based on the training set, the target policy network and the target value network in the deep neural network, determine the second loss function value of the main value network in the deep neural network, and update the network parameters of the main value network according to the second loss function value.

[0102] 6) Softly update the target policy network and the target value network. Update the network parameters of the target policy network in the deep neural network according to the updated network parameters of the main policy network, and update the network parameters of the target value network in the deep neural network according to the updated network parameters of the main value network, that is, the network parameters of the target policy network and the target value network gradually approach the network parameters of the main policy network and the main value network, as follows:

[0103]

[0104] wherein, is the network parameter of the target network (target policy network / target value network), is the network parameter of the main network (main policy network / main value network), is a hyperparameter less than 1, used to control the speed of updating.

[0105] 7) Repeat steps 2)-6) until the training stop condition is met, that is, the network model converges. Update the target policy network and the target value network as the target policy network and the target value network, respectively, return to step 2) and continue model training until the training stop condition is met, and the current main policy network is used as the reinforcement learning model.

[0106] ​In the embodiments of the present application, the main policy network and the main value network are updated based on the training set, and the target policy network and the target value network are used to provide smooth and stable training targets, and then the main network parameters are used to soft update the target network, thereby constructing a stable closed-loop training mechanism, so that the finally generated reinforcement learning model can quickly and stably converge, improve the accuracy and effectiveness of the cache optimization action, and significantly improve the database performance and system throughput.

[0107] In an exemplary embodiment, determining the first loss function value of the main policy network in the deep neural network based on the training set comprises: inputting the training set into the deep neural network, extracting each experience sample in the training set according to the main policy network of the deep neural network to obtain a cache optimization strategy corresponding to each experience sample; determining a first value of the cache optimization strategy corresponding to each experience sample according to the main value network of the deep neural network; and determining the first loss function value according to the first value of the cache optimization strategy corresponding to each experience sample.

[0108] In some embodiments, the calculation formula of the first loss function value of the main policy network is as follows:

[0109]

[0110] wherein L is the first loss function value of the main policy network, is the number of experience samples in the training set, and are network parameters of the main policy network and the main value network, respectively, is the cache optimization strategy corresponding to each experience sample, is the first value of the cache optimization strategy corresponding to each experience sample. The Q value corresponding to the action selected by the main policy network is maximized. The network parameters of the main policy network are updated using the gradient descent method, and the formula is as follows:

[0111]

[0112] wherein is the learning rate of the main policy network, is the gradient direction of the first loss function to the parameters.

[0113] In the embodiments of the present application, the cache optimization strategy is extracted from the experience sample by using the main policy network, and the strategy is evaluated by the main value network, and then the loss function is calculated based on the value, so that the strategy learning process has a clear optimization direction and stable value benchmark, thereby significantly improving the accuracy, stability and long-term performance of the cache optimization strategy.

[0114] In an exemplary embodiment, determining the second loss function value of the main value network in the deep neural network based on the training set includes: for each experience sample in the training set, extracting the next experience sample of the experience sample according to the target policy network of the deep neural network to obtain the caching optimization strategy of the next experience sample, and determining the first value of the caching optimization strategy of the next experience sample according to the target value network of the deep neural network; obtaining the reward value corresponding to the experience sample, determining the target value according to the reward value and the first value of the caching optimization strategy of the next experience sample; and determining the second loss function value according to the first value of the caching optimization strategy corresponding to each experience sample and the target value.

[0115] In some implementations, the target value of the main value network is calculated using the following formula:

[0116]

[0117] Where y is the target value, and r is the reward value corresponding to the experience sample. It is a discount factor. The next state is... It is the output of the target policy network, namely the caching optimization strategy for the next empirical sample. It is a noise term.

[0118] The formula for calculating the value of the second loss function is as follows:

[0119]

[0120] in, This is the value of the second loss function. For target value, The first value of caching optimization strategies for the next empirical sample.

[0121] The parameters of the main value network are updated using gradient descent, as shown in the following formula:

[0122]

[0123] in, The learning rate of the main value network, This represents the gradient direction of the second loss function with respect to the parameters.

[0124] In this embodiment, the next empirical sample is extracted and its value is calculated by using a target policy network and a target value network. A stable target value is constructed by combining the reward value. Then, the second loss function value of the main value network is calculated based on the target value and the current value. This can significantly improve the stability, accuracy and convergence speed of the main value network training, thereby effectively improving the reliability and execution performance of the cache optimization strategy generated by the reinforcement learning model.

[0125] Furthermore, real-time decision-making and query optimization can be performed: during the optimization process, based on the reinforcement learning model, the system will make real-time query adjustments according to the current database performance indicators and cache hit rate.

[0126] Performance evaluation and optimization: By regularly evaluating and optimizing the query efficiency of the database, new historical data can be collected and used to update the reinforcement learning model, thereby continuously improving the tuning capabilities of the cache optimizer.

[0127] Through the detailed steps described above, the reinforcement learning-based cache optimizer can optimize the query and scheduling of database system caches, thereby improving database performance and efficiency.

[0128] In an exemplary embodiment, the SQL query execution process in a reinforcement learning scenario is illustrated with an example, such as... Figure 4 As shown. Suppose there is an e-commerce database system running a long-term, stable MySQL instance, with a frequently accessed `orders` table containing a large amount of historical order data. Now, the system receives a complex SQL query request:

[0129] SELECT customer_id, SUM(amount); This query retrieves the customer's ID and the total amount spent by that customer.

[0130] FROM orders; retrieves data from the orders table.

[0131] WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31'; Condition: Order date is between 2024-01-01 and 2024-03-31;

[0132] GROUP BY customer_id; group by customer ID;

[0133] ORDER BY SUM(amount) DESC; Sort by total consumption in descending order (from highest to lowest).

[0134] LIMIT 10; retrieves only the first 10 records.

[0135] Traditional execution process (without reinforcement learning cache optimizer):

[0136] 1. Query parsing and optimization: The MySQL optimizer analyzes the query and decides to use the order_date index of the orders table for a range scan.

[0137] 2. Data reading: Due to the large amount of data and the possibility of not caching the required data pages in the Buffer Pool, the system needs to read a large number of data blocks from the disk.

[0138] 3. Calculation and sorting: Aggregate calculation and sorting on the read data, which may involve temporary disk files due to the large amount of data.

[0139] 4. Result return: Return the top 10 records.

[0140] 5. Problem: If this query is frequently executed during peak hours and the required data pages are not hit in the Buffer Pool, it will cause a large number of disk I / O, increase query delay, and reduce system throughput.

[0141] Execution process after using the reinforcement learning cache optimizer of the present application:

[0142] 1. State awareness and action decision:

[0143] a. The reinforcement learning model monitors the database state in real time, including the pages cached in the current Buffer Pool (represented by a bitmap), the query queue, historical access patterns, etc.

[0144] b. When the above SQL query enters the queue, the optimizer compares its features (such as accessed tables, indexes, data range) with the current cache state.

[0145] c. The Actor network outputs action a according to the current state s (such as the cache bitmap showing that the data block of the orders table in Q1 2024 is low in the cache ratio) : decides to prioritize retaining or preloading the data block of the orders table in Q1 2024 to the Buffer Pool.

[0146] 2. Cache preloading and hit optimization:

[0147] a. The system loads the data block of the orders table from January to March 2024 (such as through asynchronous I / O or adjusting the cache replacement strategy) into the Buffer Pool in advance according to the policy network suggestion.

[0148] b. When the query is executed, the required data page is already in memory, the cache hit rate is significantly improved, and disk I / O is avoided.

[0149] 3. Reward feedback and policy update:

[0150] a. After query execution, the system records performance indicators: throughput is increased by 15%, cache hit rate is increased from 60% to 85%, and query response time is reduced from 2.3 seconds to 0.8 seconds.

[0151] b. The reward function calculates positive reward (r>0) because the throughput and cache hit rate are higher than the baseline.

[0152] c. The experience sample (state, action, reward value, next state) is stored in a priority experience replay buffer for subsequent training.

[0153] d. The Critic network evaluates the action value through TD error, and the Actor network updates the policy to reinforce the behavior of "preloading high-frequency access time period data".

[0154] 4. Long-term policy optimization:

[0155] a. As more similar queries are executed, the model learns that at the beginning of the month / quarter, the recent order data block should be prioritized for retention.

[0156] b. The system gradually forms a long-term policy: even if there are no relevant queries in the short term, a certain proportion of the latest order data is maintained in the cache to respond to future analysis requests.

[0157] Through the cache optimizer based on reinforcement learning proposed in the present application, the database system can intelligently perceive query patterns and data access hotspots; dynamically adjust the cache content to achieve "store by query"; balance between short-term response and long-term performance; continuously optimize itself to adapt to changes in business load. The example fully demonstrates how the reinforcement learning model can transform the originally inefficient SQL execution process into an efficient and adaptive cache utilization process through state representation, action decision, and reward feedback mechanism, significantly improving database I / O efficiency and overall performance (cache hit rate, query response time, IO disk times, system QPS (Queries Per Second, the number of queries / requests that the system can handle per second)), verifying the practicality and advancement of the present application.

[0158] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0159] Based on the same inventive concept, the embodiments of the present application also provide a database cache optimization apparatus for implementing the database cache optimization method described above. The implementation scheme of the apparatus for solving the problem is similar to the implementation scheme described in the above method, so the specific limitations in one or more database cache optimization apparatus embodiments provided below can refer to the limitations of the database cache optimization method described above, which will not be repeated here.

[0160] In one exemplary embodiment, as shown in Figure 5 A database cache optimization apparatus is provided, comprising: a definition module 502, a prediction module 504 and an optimization module 506, wherein:

[0161] The definition module 502 is configured to perform state representation on the current cache state parameters of the database cache to obtain current state representation data, and determine a current cache optimization space according to the current state representation data.

[0162] The prediction module 504 is configured to determine a current cache optimization action according to the reinforcement learning model, the current state representation data and the current cache optimization space.

[0163] The optimization module 506 is configured to optimize the database cache according to the current cache optimization action.

[0164] In one exemplary embodiment, the prediction module 504 is further configured to extract a current state vector from the current state representation data; input the current state vector and the space information of the current cache optimization space into the reinforcement learning model, and determine the current cache optimization action in the current cache optimization space by a main policy network in the reinforcement learning model.

[0165] In one exemplary embodiment, the apparatus further comprises:

[0166] A loop module is configured to obtain target cache state parameters corresponding to the optimized data cache as next cache state parameters.

[0167] The definition module 502 is further configured to perform state representation on the target cache state parameters to obtain target state representation data, and determine a target cache optimization space according to the target state representation data.

[0168] The prediction module 504 is further configured to determine a target cache optimization action according to the reinforcement learning model, the target state representation data and the target cache optimization space.

[0169] The optimization module 506 is further configured to re-optimize the optimized database cache according to the target cache optimization action.

[0170] In one exemplary embodiment, the apparatus further comprises:

[0171] The database performance evaluation module is configured to obtain database performance feedback data corresponding to the optimized data cache, and evaluate the performance of the database corresponding to the optimized data cache according to the database performance feedback data.

[0172] In an exemplary embodiment, the apparatus further comprises:

[0173] The model training module is configured to determine a first loss function value of the main policy network in the deep neural network based on the training set, and update network parameters of the main policy network according to the first loss function value; determine a second loss function value of the main value network in the deep neural network based on the training set, the target policy network and the target value network in the deep neural network, and update network parameters of the main value network according to the second loss function value; update network parameters of the target policy network in the deep neural network according to the updated network parameters of the main policy network, and update network parameters of the target value network in the deep neural network according to the updated network parameters of the main value network; return to the step of determining the first loss function value of the main policy network in the deep neural network based on the training set, continue the model training until a training stop condition is met, and take the current main policy network as the reinforcement learning model.

[0174] In an exemplary embodiment, the model training module is further configured to, for each experience sample in the pre-created experience replay buffer, obtain a first value, a second value and a reward value corresponding to the experience sample, and determine a time difference error of the experience sample according to the first value, the second value and the reward value corresponding to the experience sample; wherein the first value is output by the main value network, and the second value is output by the target value network; determine a sample sampling probability according to the time difference error; and select the training set from the experience replay buffer according to the sample sampling probability.

[0175] In an exemplary embodiment, the model training module is further configured to input the training set into the deep neural network, extract and process each experience sample in the training set according to the main policy network of the deep neural network, to obtain a cache optimization strategy corresponding to each experience sample; determine a first value of the cache optimization strategy corresponding to each experience sample according to the main value network of the deep neural network; and determine the first loss function value according to the first value corresponding to the cache optimization strategy of each experience sample.

[0176] In an example embodiment, the model training module is further configured to, for each experience sample in the training set, extract a next experience sample of the experience sample according to a target policy network of the deep neural network, to obtain a cache optimization strategy of the next experience sample of the experience sample, and determine a first value of the cache optimization strategy of the next experience sample according to a target value network of the deep neural network; obtain a reward value corresponding to the experience sample, and determine a target value according to the reward value and the first value of the cache optimization strategy of the next experience sample; and determine a second loss function value according to the first value of the cache optimization strategy of each experience sample and the target value.

[0177] Each module in the database cache optimization apparatus can be implemented wholly or partially by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0178] In an example embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 6 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals, and the wireless communication can be achieved through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a database cache optimization method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, a trackball, or a touchpad arranged on the shell of the computer device, or can be an external keyboard, a touchpad, a mouse, or the like.

[0179] Those skilled in the art can understand that Figure 6The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0181] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0182] In an embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0185] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0186] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A database caching optimization method, characterized in that, The method includes: The current cache status parameters of the database cache are represented to obtain the current status representation data, and the current cache optimization space is determined based on the current status representation data; Based on the reinforcement learning model, the current state representation data, and the current cache optimization space, determine the current cache optimization action; The database cache is optimized based on the current cache optimization action.

2. The method according to claim 1, characterized in that, The step of determining the current cache optimization action based on the reinforcement learning model, the current state representation data, and the current cache optimization space includes: Extract the current state vector from the current state representation data; The current state vector and the spatial information of the current cache optimization space are input into the reinforcement learning model, and the main policy network in the reinforcement learning model determines the current cache optimization action in the current cache optimization space.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the target cache state parameter corresponding to the optimized data cache, and use it as the next cache state parameter; The target cache state parameters are represented to obtain target state representation data, and the target cache optimization space is determined based on the target state representation data. Based on the reinforcement learning model, the target state representation data, and the target cache optimization space, determine the target cache optimization action; The optimized database cache is further optimized based on the target cache optimization action.

4. The method according to claim 1, characterized in that, The method further includes: Retrieve the database performance feedback data corresponding to the optimized data cache; The performance of the database corresponding to the optimized database cache is evaluated based on the database performance feedback data.

5. The method according to claim 1, characterized in that, The method further includes: The first loss function value of the main policy network in the deep neural network is determined based on the training set, and the network parameters of the main policy network are updated according to the first loss function value. Based on the training set, the target policy network and the target value network in the deep neural network, the second loss function value of the main value network in the deep neural network is determined, and the network parameters of the main value network are updated according to the second loss function value. The network parameters of the target policy network in the deep neural network are updated based on the updated network parameters of the master policy network, and the network parameters of the target value network in the deep neural network are updated based on the updated network parameters of the master value network. The updated target policy network and the updated target value network are used as the target policy network and the target value network, respectively. The step of determining the first loss function value of the master policy network in the deep neural network based on the training set is returned, and model training continues until the training stopping condition is met. The current master policy network is used as the reinforcement learning model.

6. The method according to claim 5, characterized in that, The method further includes: For each experience sample in the pre-created experience replay buffer, the first value, second value, and reward value corresponding to the experience sample are obtained, and the time difference error of the experience sample is determined based on the first value, second value, and reward value corresponding to the experience sample; wherein, the first value is output by the main value network, and the second value is output by the target value network; The sample sampling probability is determined based on the time difference error. The training set is selected from the experience replay buffer according to the sample sampling probability.

7. The method according to claim 5, characterized in that, The step of determining the first loss function value of the main policy network in the deep neural network based on the training set includes: The training set is input into the deep neural network, and the experience samples in the training set are extracted and processed according to the main policy network of the deep neural network to obtain the cache optimization strategy corresponding to each experience sample. The first value of the cache optimization strategy corresponding to each empirical sample is determined based on the main value network of the deep neural network. The first loss function value is determined based on the first value corresponding to the cache optimization strategy for each empirical sample.

8. The method according to claim 7, characterized in that, Determining the second loss function value of the main value network in the deep neural network based on the training set includes: For each experience sample in the training set, the next experience sample of the experience sample is extracted and processed according to the target policy network of the deep neural network to obtain the caching optimization strategy of the next experience sample of the experience sample, and the first value of the caching optimization strategy of the next experience sample is determined according to the target value network of the deep neural network. Obtain the reward value corresponding to the experience sample, and determine the target value based on the reward value and the first value of the caching optimization strategy for the next experience sample; The second loss function value is determined based on the first value of the cache optimization strategy corresponding to each empirical sample and the target value.

9. A database caching optimization device, characterized in that, The device includes: The definition module is used to represent the current cache status parameters of the database cache, obtain the current status representation data, and determine the current cache optimization space based on the current status representation data; The prediction module is used to determine the current cache optimization action based on the reinforcement learning model, the current state representation data, and the current cache optimization space. The optimization module is used to optimize the database cache based on the current cache optimization action.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images