A cache optimization management method, device and equipment and readable storage medium

Abstract

The application discloses a cache optimization management method and device, equipment and a readable storage medium, and belongs to the field of computer equipment, and is used for optimizing and managing the memory pool of a computer equipment. Considering that some data tables in a disk are frequently accessed, the data tables need to be repeatedly loaded from the disk to the memory pool each time, and time is wasted. Therefore, the application can determine the to-be-solidified data table that needs to be frequently accessed from the two aspects of the data table corresponding to the data access task that is accessed with a heat value greater than a second preset threshold and is executed in a large amount, and then store the to-be-solidified data table in a first memory pool. When the data table is frequently accessed, the time for loading the data table from the disk to the memory pool is greatly saved, the work efficiency is improved, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of computer equipment, and in particular to a cache optimization management method. This invention also relates to a cache optimization management device, apparatus, and computer-readable storage medium. Background Technology

[0002] The typical process for computer devices to perform data access tasks is to load the data table from the disk into the memory pool, and then retrieve the data table from the memory pool for related calculations. This complete process takes a long time. Combined with the increasing volume of data access, this results in poor efficiency of existing data access processes, which affects the user experience.

[0003] Therefore, how to provide a solution to the above-mentioned technical problems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a cache optimization management method that identifies frequently accessed data tables to be persisted based on two criteria: data tables whose access frequency exceeds a second preset threshold and data tables corresponding to frequently executed data access tasks. These tables are then persistently stored in a first memory pool. This significantly reduces the time required to load these data tables from disk to the memory pool when they are frequently accessed, improving work efficiency and enhancing user experience. Another purpose of this invention is to provide a cache optimization management device, equipment, and computer-readable storage medium that identifies frequently accessed data tables to be persisted based on two criteria: data tables whose access frequency exceeds a second preset threshold and data tables corresponding to frequently executed data access tasks. These tables are then persistently stored in a first memory pool. This significantly reduces the time required to load these data tables from disk to the memory pool when they are frequently accessed, improving work efficiency and enhancing user experience.

[0005] To address the aforementioned technical problems, this invention provides a cache optimization management method, comprising:

[0006] The statistics include the number of times each data access task is executed and the access frequency of each data table on the disk within a preset time period based on the current time.

[0007] The data table accessed by the data access task whose execution count exceeds a first preset threshold is designated as the data table to be solidified.

[0008] The data table whose access popularity value is greater than the second preset threshold is designated as the data table to be solidified;

[0009] Each of the data tables to be solidified is persistently stored in the first memory pool so that each of the data tables to be solidified can be directly accessed from the first memory pool;

[0010] The access popularity value of each data table is positively correlated with the number of times the data table is accessed.

[0011] Preferably, after calculating the execution count of each data access task and the access frequency of each data table on the disk within a preset time period based on the current time, the cache optimization management method further includes:

[0012] The popularity level to which the access popularity value of each data table belongs is determined according to the preset popularity level range;

[0013] After designating the data tables whose access popularity value is greater than a second preset threshold as data tables to be cached, the cache optimization management method further includes:

[0014] Allocate additional memory space from the current remaining capacity of the first memory pool;

[0015] According to the order of popularity from high to low, each of the data tables is persistently stored in the additional memory space.

[0016] Preferably, after calculating the execution count of each data access task and the access frequency of each data table on the disk within a preset time period based on the current time, the cache optimization management method further includes:

[0017] After the data access task with an execution count greater than a first preset threshold is executed for the first time, the calculation result data generated by the execution is persistently stored in the first memory pool.

[0018] Preferably, after calculating the execution count of each data access task and the access frequency of each data table on the disk within a preset time period based on the current time, the cache optimization management method further includes:

[0019] Add a removal tag to the data table whose access popularity value is lower than the third preset threshold;

[0020] Determine whether the removed tag exists in the data table that is persistently stored in the first memory pool;

[0021] If it exists, the data table will be removed from the first memory pool after it has been used up.

[0022] Preferably, the computer device further includes a second memory pool;

[0023] This cache optimization management method also includes:

[0024] Every preset period, it is determined whether the remaining capacity in the first memory pool is less than a fourth preset threshold.

[0025] If it is less than, then select the task to be migrated from the tasks to be executed in the first memory pool;

[0026] The task to be migrated is added to the task queue of the second memory pool so that the second memory pool can load the data corresponding to the task to be migrated.

[0027] Preferably, the cache optimization management method further includes:

[0028] Determine whether the memory required by the next data access task to be executed in the task queue exceeds the remaining capacity of its corresponding memory pool;

[0029] If the limit is exceeded, the execution of the task queue containing the pending data access task will be suspended.

[0030] When it is detected that the remaining capacity of the memory pool corresponding to the data access task to be executed is sufficient to meet the memory capacity required by the data access task to be executed, the tasks in the task queue continue to be executed.

[0031] Preferably, before persistently storing each of the data tables to be solidified in the first memory pool so that each of the data tables to be solidified can be directly accessed from the first memory pool, the cache optimization management method further includes:

[0032] Receive the specified data table to be solidified through a human-computer interaction device.

[0033] To address the aforementioned technical problems, the present invention provides a cache optimization management device, comprising:

[0034] The statistics module is used to count the number of times each data access task is executed and the access frequency value of each data table on the disk within a preset time period based on the current time.

[0035] The first analysis module is used to identify the data table accessed by the data access task whose execution count exceeds a first preset threshold as the data table to be solidified.

[0036] The second analysis module is used to identify data tables whose access popularity value is greater than a second preset threshold as data tables to be solidified.

[0037] An execution module is used to persistently store each of the data tables to be solidified in a first memory pool so that each of the data tables to be solidified can be directly accessed from the first memory pool.

[0038] The access popularity value of each data table is positively correlated with the number of times the data table is accessed.

[0039] To address the aforementioned technical problems, this invention provides a cache optimization management device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor for implementing the cache optimization management method as described above when executing the computer program.

[0042] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cache optimization management method described above.

[0043] This invention provides a cache optimization management method. Considering that some data tables on the disk are frequently accessed, these data tables need to be repeatedly loaded from the disk into the memory pool each time, wasting time. Therefore, this application can determine the data tables that need to be frequently accessed and need to be persisted from two perspectives: the access popularity value is greater than a second preset threshold and the data tables corresponding to the data access tasks that are executed in large numbers. Then, these tables are persistently stored in the first memory pool. When these data tables are frequently accessed, the time spent loading from the disk into the memory pool is greatly reduced, improving work efficiency and enhancing user experience.

[0044] The present invention also provides a cache optimization management device, apparatus, and computer-readable storage medium, which have the same beneficial effects as the cache optimization management method described above. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a cache optimization management method provided by the present invention;

[0047] Figure 2 A schematic diagram of the structure of a cache optimization management device provided by the present invention;

[0048] Figure 3 This is a schematic diagram of a cache optimization management device provided by the present invention. Detailed Implementation

[0049] The core of this invention is to provide a cache optimization management method. This method identifies frequently accessed data tables that need to be persisted by considering two factors: data tables whose access frequency exceeds a second preset threshold and data tables corresponding to frequently executed data access tasks. These tables are then persistently stored in a first memory pool. This significantly reduces the time required to load these data tables from disk to the memory pool when they are frequently accessed, improving work efficiency and enhancing user experience. Another core aspect of this invention is to provide a cache optimization management device, equipment, and computer-readable storage medium. This method identifies frequently accessed data tables that need to be persisted by considering two factors: data tables whose access frequency exceeds a second preset threshold and data tables corresponding to frequently executed data access tasks. These tables are then persistently stored in a first memory pool. This significantly reduces the time required to load these data tables from disk to the memory pool when they are frequently accessed, improving work efficiency and enhancing user experience.

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a cache optimization management method provided by the present invention. The cache optimization management method includes:

[0052] S101: Statistically count the number of times each data access task is executed and the access popularity value of each data table on the disk within a preset time period based on the current time.

[0053] Specifically, considering the technical problems mentioned in the background, and taking into account that some data tables on the disk are frequently accessed, these data tables need to be repeatedly loaded from the disk into the memory pool each time, wasting time. Therefore, the applicant wants to find these frequently accessed data tables and then persistently store them in the memory pool to save the time of loading data into memory. Considering that whether a data table is frequently accessed can be judged by the concept of "access popularity", and that for some data access tasks that are executed in large numbers, the accessed data tables also need to be frequently loaded into memory, this application intends to first find "frequently accessed" data tables from these two perspectives. This step can be statistically analyzed periodically, with a preset time period in the past, to count the number of executions of each data access task and the access popularity value of each data table on the disk, and use this as the data basis for subsequent steps.

[0054] The essence of the access popularity value is the number of times the data table is accessed within a certain period of time. A higher number of accesses results in a higher popularity value, and vice versa. The calculation method for the access popularity value can be customized. For example, it can be: 5A + 3B + 2C = H, where A is the number of reads of the data table in the past preset period of time, B is the number of writes of the data table in the past preset period of time, C is the number of deletes of the data table in the past preset period of time, and H is the access popularity value of the data table.

[0055] Specifically, the preset duration can be set independently, for example, it can be 1 week, etc., but this embodiment of the invention does not limit it here.

[0056] Specifically, a data table can be understood as a collection of data, that is, a series of data.

[0057] S102: The data table accessed by the data access task whose execution count exceeds the first preset threshold is used as the data table to be solidified;

[0058] Specifically, based on the above theory, data tables accessed frequently by data access tasks that have been executed a lot within a preset time period will be accessed frequently. To determine whether a task has been "executed a lot", this embodiment of the invention sets a first preset threshold. The number of executions is compared with the first preset threshold. If the number of executions is greater than the first preset threshold, it can be considered that the task has been "executed a lot", and the corresponding data table will be used as a data table to be solidified, so as to serve as the data foundation for subsequent steps.

[0059] The first preset threshold can be set independently, and this embodiment of the invention does not limit it.

[0060] S103: Data tables whose access popularity value is greater than the second preset threshold are designated as data tables to be solidified;

[0061] Specifically, similar to the logic in S102, in order to determine the data table that is "frequently accessed", this embodiment of the invention sets a second preset threshold for the access popularity value and compares the access popularity value with the second preset threshold. As long as the access popularity value is greater than the second preset threshold, the data table can be used as a data table to be solidified, so as to serve as the data basis for subsequent steps.

[0062] The second preset threshold can be set independently, and this embodiment of the invention does not limit it.

[0063] S104: Persistently store each data table to be solidified in the first memory pool so that each data table to be solidified can be accessed directly from the first memory pool;

[0064] The access popularity value of each data table is positively correlated with the number of times the data table is accessed.

[0065] Specifically, based on the data tables to be solidified identified in the above steps, these data tables can be persistently stored in the first memory pool. In this way, when these data tables are frequently accessed in the future, the process of loading these data tables from the disk to the memory pool is eliminated. Instead, these data tables can be directly obtained from the memory pool for direct use, making reasonable use of the memory pool space, improving work efficiency and enhancing user experience.

[0066] It is worth mentioning that the preset duration, the first preset threshold, and the second preset threshold can all be configured and modified at any time.

[0067] Additionally, it is worth mentioning that the embodiments of the present invention can be applied to Presto, a memory-based parallel computing framework and a distributed SQL (Structured Query Language) interactive query engine. Its working mode is as follows: when a query task is obtained, the SQL is first parsed into an abstract syntax tree, then a specific query plan is obtained, and finally, it is distributed parsed into individual tasks and distributed to cluster nodes for plan execution and calculation. Specifically, in a single node, i.e., the worker process, the plan is executed, and its main work is to load data and perform logical calculations according to the task content of the plan, and return the corresponding results.

[0068] Of course, besides Presto, the cache optimization management method in this embodiment of the invention can also be applied to other places, and this embodiment of the invention does not limit it here.

[0069] This invention provides a cache optimization management method. Considering that some data tables on the disk are frequently accessed, these data tables need to be repeatedly loaded from the disk into the memory pool each time, wasting time. Therefore, this application can determine the data tables that need to be frequently accessed and need to be persisted from two perspectives: the access popularity value is greater than a second preset threshold and the data tables corresponding to the data access tasks that are executed in large numbers. Then, these tables are persistently stored in the first memory pool. When these data tables are frequently accessed, the time spent loading from the disk into the memory pool is greatly reduced, improving work efficiency and enhancing user experience.

[0070] Based on the above embodiments:

[0071] As a preferred embodiment, after statistically analyzing the execution count of each data access task and the access frequency of each data table on the disk over a preset time period based on the current time, the cache optimization management method further includes:

[0072] The popularity level of each data table is determined based on the preset popularity level range.

[0073] After designating data tables with access popularity values ​​exceeding a second preset threshold as tables to be cached, this cache optimization management method also includes:

[0074] Allocate additional memory space from the current remaining capacity of the first memory pool;

[0075] The data tables are persistently stored in additional memory space, ordered from highest to lowest popularity.

[0076] Specifically, considering that in some cases, after some data tables are persisted in the first memory pool in the above manner, the number of data tables still stored persistently is still relatively small, that is, the remaining capacity in the first memory pool is still relatively large, which is undoubtedly a waste of resources. Therefore, in order to make reasonable use of this part of the resources, in this embodiment of the invention, the data tables are persistently stored in the first memory pool in order of access popularity from high to low. Of course, here only the extra memory space can be allocated from the current remaining capacity of the first memory pool to store this part of the data tables, so as to ensure that the first memory pool has enough capacity to meet the needs of normal operation.

[0077] Specifically, the process of allocating additional memory space from the current remaining capacity of the first memory pool can be as follows: determine whether the current remaining capacity of the first memory pool is greater than the preset demand; if it is, then 10% of the remaining capacity can be allocated as additional memory space.

[0078] Specifically, the preset heat level range can also be set independently. For example, if the preset duration is one day, it can be set that H greater than 1000 is hot, 500-1000 is warm, and below 500 is cold, for a total of three levels. This embodiment of the invention does not limit the specific heat level.

[0079] As a preferred embodiment, after statistically analyzing the execution count of each data access task and the access frequency of each data table on the disk over a preset time period based on the current time, the cache optimization management method further includes:

[0080] After a data access task with more than one preset threshold number of executions is executed for the first time, the resulting calculation data will be persistently stored in the first memory pool.

[0081] Specifically, considering that the data access task will generate calculation results after execution, and that the difference in the calculation results is not significant in each execution of the data access task, the calculation results of frequently executed data access tasks can be persistently stored in the first memory pool in this embodiment of the invention. Then, when the data access task is not executed for the first time, the calculation results in the first memory pool can be directly used, saving calculation time and improving work efficiency and user experience.

[0082] As a preferred embodiment, after statistically analyzing the execution count of each data access task and the access frequency of each data table on the disk over a preset time period based on the current time, the cache optimization management method further includes:

[0083] Data tables whose access popularity value is lower than the third preset threshold will be marked with a removal tag;

[0084] Determine if the accessed data table, which is persistently stored in the first memory pool, has a removal tag;

[0085] If it exists, the data table will be removed from the first memory pool after it has been used up.

[0086] Specifically, considering that the access popularity of a data table may decrease over time, if the access popularity of a data table persistently stored in the first memory pool is low enough, it is no longer suitable to continue storing it in the first memory pool. Therefore, this embodiment of the invention can mark a data table with an access popularity value lower than a third preset threshold as a removal tag, and determine whether the accessed data table persistently stored in the first memory pool has a removal tag. If a removal tag exists, the data table can be removed from the first memory pool after it has been used. In the subsequent loading process, it can also be determined whether the currently accessed data table has a removal tag. If it does, the data table needs to be released from the first memory pool after it has been accessed.

[0087] The third preset threshold can be set independently, and this embodiment of the invention does not limit it.

[0088] Specifically, the popularity levels and data to be solidified mentioned above can be tagged. For example, when a data table contains data to be solidified, it can be tagged with "solidified".

[0089] In a preferred embodiment, the computer device further includes a second memory pool;

[0090] This cache optimization management method also includes:

[0091] Every preset period, determine whether the remaining capacity in the first memory pool is less than the fourth preset threshold;

[0092] If it is less than, then select the task to be migrated from the tasks to be executed in the first memory pool;

[0093] Add the tasks to be migrated to the task queue of the second memory pool so that the second memory pool can load the data corresponding to the tasks to be migrated.

[0094] Specifically, considering the possibility that the first memory pool may be filled with a large number of data tables during operation, leading to operational abnormalities, this embodiment of the invention designs a second memory pool and sets an overflow mechanism for the first memory pool. That is, every preset period, it can be determined whether the remaining capacity in the first memory pool is less than a fourth preset threshold. If it is less, it means that the first memory pool is at risk of being filled. Then, a task to be migrated can be selected from the tasks to be executed in the first memory pool and added to the task queue of the second memory pool so that the second memory pool can load the data corresponding to the task to be migrated, thereby reducing the pressure on the first memory pool and reducing the possibility of it becoming abnormal.

[0095] There are various ways to select the task to be migrated from the tasks to be executed in the first memory pool. These methods can be set by the user. For example, the task with the largest amount of data in the tasks to be executed in the first memory pool can be selected as the task to be migrated. This embodiment of the invention does not limit these methods.

[0096] Specifically, before identifying the tasks to be migrated, it is also necessary to ensure that the second memory pool currently has enough space to receive new tasks.

[0097] As a preferred embodiment, the cache optimization management method further includes:

[0098] Determine whether the memory required by the next data access task to be executed in the task queue exceeds the remaining capacity of its corresponding memory pool;

[0099] If the limit is exceeded, the task queue containing the pending data access task will be suspended.

[0100] When it is detected that the remaining capacity of the memory pool corresponding to the data access task to be executed is sufficient to meet the memory capacity required by the data access task to be executed, the tasks in the task queue continue to be executed.

[0101] Specifically, considering that if the remaining capacity in the current memory pool is insufficient when executing a massive computing task, the task will fail, this embodiment of the invention can determine whether the memory capacity required by the next data access task to be executed in the task queue exceeds the remaining capacity of its corresponding memory pool. If it does, the execution of the task queue containing the data access task to be executed can be suspended until the remaining capacity of the memory pool corresponding to the data access task to be executed is detected to meet the memory capacity required by the data access task to be executed. Then, the tasks in the task queue can continue to be executed, so that the remaining capacity of the memory pool can meet the needs of the massive computing task, prevent task failure, and improve the user experience.

[0102] As a preferred embodiment, before directly accessing each data table to be persisted in a first memory pool, the cache optimization management method further includes:

[0103] Receive the specified data table to be solidified through a human-computer interaction device.

[0104] Specifically, considering that the data table to be solidified determined by the above-mentioned automated method cannot meet the intuitive needs of users, in this embodiment of the invention, a specified data table to be solidified can be received through a human-computer interaction device so as to persistently store the user-specified data table in the first memory pool, thereby further meeting user needs and improving user experience.

[0105] The human-computer interaction device can be of various types, such as a keyboard and mouse, and this embodiment of the invention does not limit it.

[0106] In addition, it is worth mentioning that, considering the risk of insufficient remaining capacity in each memory pool, this embodiment of the invention can monitor the remaining capacity in each memory pool and control the alarm to sound when the remaining capacity drops below the warning value, so that staff can handle the situation in a timely manner and avoid abnormalities.

[0107] Please refer to Figure 2 , Figure 2 The present invention provides a cache optimization management device, which includes:

[0108] The statistics module 21 is used to count the number of times each data access task is executed and the access popularity value of each data table in the disk within a preset time period based on the current time.

[0109] The first analysis module 22 is used to select the data table accessed by the data access task whose execution count is greater than the first preset threshold as the data table to be solidified.

[0110] The second analysis module 23 is used to treat data tables whose access popularity value is greater than the second preset threshold as data tables to be solidified.

[0111] The execution module 24 is used to persistently store each data table to be solidified in the first memory pool so that each data table to be solidified can be directly accessed from the first memory pool.

[0112] The access popularity value of each data table is positively correlated with the number of times the data table is accessed.

[0113] This invention provides a cache optimization management method. Considering that some data tables on the disk are frequently accessed, these data tables need to be repeatedly loaded from the disk into the memory pool each time, wasting time. Therefore, this application can determine the data tables that need to be frequently accessed and need to be persisted from two perspectives: the access popularity value is greater than a second preset threshold and the data tables corresponding to the data access tasks that are executed in large numbers. Then, these tables are persistently stored in the first memory pool. When these data tables are frequently accessed, the time spent loading from the disk into the memory pool is greatly reduced, improving work efficiency and enhancing user experience.

[0114] In a preferred embodiment, the cache optimization management device further includes:

[0115] The first calculation module is used to determine the popularity level of the accessed popularity value of each data table according to the preset popularity level range.

[0116] The partitioning module is used to allocate additional memory space from the current remaining capacity of the first memory pool;

[0117] The first action module is used to persistently store each data table in an additional memory space according to the order of popularity from high to low.

[0118] In a preferred embodiment, the cache optimization management device further includes:

[0119] The second action module is used to persistently store the calculation results in the first memory pool after a data access task with more than one preset threshold number of executions is executed for the first time.

[0120] In a preferred embodiment, the cache optimization management device further includes:

[0121] The third module is used to tag data tables whose access popularity value is lower than the third preset threshold with a removal tag;

[0122] The first judgment module is used to determine whether the accessed data table, which is persistently stored in the first memory pool, has a removal tag. If it does, the clearing module is triggered.

[0123] The clearing module is used to clear a data table from the first memory pool after it has been used up.

[0124] In a preferred embodiment, the computer device further includes a second memory pool;

[0125] The cache optimization management device also includes:

[0126] The second judgment module is used to determine whether the remaining capacity in the first memory pool is less than the fourth preset threshold every preset period. If it is less, the selection module is triggered.

[0127] The selection module is used to select tasks to be migrated from the tasks to be executed in the first memory pool;

[0128] The third action module is used to add the task to be migrated to the task queue of the second memory pool so that the second memory pool can load the data corresponding to the task to be migrated.

[0129] In a preferred embodiment, the cache optimization management device further includes:

[0130] The third judgment module is used to determine whether the memory capacity required by the next data access task to be executed in the task queue exceeds the remaining capacity of its corresponding memory pool. If it does, the fourth action module is triggered.

[0131] The fourth action module is used to pause the execution of the task queue containing the pending data access tasks;

[0132] The fifth action module is used to continue executing tasks in the task queue when it is detected that the remaining capacity of the memory pool corresponding to the data access task to be executed meets the memory capacity required by the data access task to be executed.

[0133] In a preferred embodiment, the cache optimization management device further includes:

[0134] The receiving module is used to receive a specified data table to be solidified through a human-computer interaction device.

[0135] For a description of the cache optimization management device provided in the embodiments of the present invention, please refer to the aforementioned embodiments of the cache optimization management method; the embodiments of the present invention will not be repeated here.

[0136] Please refer to Figure 3 , Figure 3 The present invention provides a cache optimization management device, which includes:

[0137] Memory 31 is used to store computer programs;

[0138] The processor 32 is used to implement the steps of the cache optimization management method as described in the foregoing embodiments when executing a computer program.

[0139] Specifically, the memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer-readable instructions, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. When the processor executes the computer program stored in the memory, it can perform the following steps: count the number of times each data access task is executed and the access popularity value of each data table in the disk within a preset time period based on the current time; designate the data tables accessed by data access tasks with an execution count greater than a first preset threshold as data tables to be solidified; designate the data tables with access popularity values ​​greater than a second preset threshold as data tables to be solidified; and persistently store each data table to be solidified in a first memory pool so that each data table to be solidified can be directly accessed from the first memory pool; wherein, the access popularity value of each data table is positively correlated with the number of times the data table is accessed.

[0140] This invention provides a cache optimization management device. Considering that some data tables on the disk are frequently accessed, these data tables need to be repeatedly loaded from the disk into the memory pool each time, wasting time. Therefore, this application can determine the data tables that need to be frequently accessed and need to be persisted from two perspectives: the access popularity value is greater than a second preset threshold and the data tables corresponding to the data access tasks that are executed in large numbers. Then, these tables are persistently stored in the first memory pool. When these data tables are frequently accessed, the time spent loading from the disk into the memory pool is greatly reduced, improving work efficiency and enhancing user experience.

[0141] As an optional embodiment, when the processor executes the computer subroutine stored in the memory, it can perform the following steps: determine the popularity level to which the access popularity value of each data table belongs based on a preset popularity level range;

[0142] Allocate additional memory space from the current remaining capacity of the first memory pool;

[0143] The data tables are persistently stored in additional memory space, ordered from highest to lowest popularity.

[0144] As an optional embodiment, when the processor executes a computer subroutine stored in memory, it may perform the following steps:

[0145] After a data access task with more than one preset threshold number of executions is executed for the first time, the resulting calculation data will be persistently stored in the first memory pool.

[0146] As an optional embodiment, when the processor executes a computer subroutine stored in memory, it may perform the following steps:

[0147] Data tables whose access popularity value is lower than the third preset threshold will be marked with a removal tag;

[0148] Determine if the accessed data table, which is persistently stored in the first memory pool, has a removal tag;

[0149] If it exists, the data table will be removed from the first memory pool after it has been used up.

[0150] As an optional embodiment, when the processor executes a computer subroutine stored in memory, it may perform the following steps:

[0151] Every preset period, determine whether the remaining capacity in the first memory pool is less than the fourth preset threshold;

[0152] If it is less than, then select the task to be migrated from the tasks to be executed in the first memory pool;

[0153] Add the tasks to be migrated to the task queue of the second memory pool so that the second memory pool can load the data corresponding to the tasks to be migrated.

[0154] As an optional embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps: determine whether the memory capacity required by the next data access task to be executed in the task queue exceeds the remaining capacity of its corresponding memory pool;

[0155] If the limit is exceeded, the task queue containing the pending data access task will be suspended.

[0156] When it is detected that the remaining capacity of the memory pool corresponding to the data access task to be executed is sufficient to meet the memory capacity required by the data access task to be executed, the tasks in the task queue continue to be executed.

[0157] As an optional embodiment, when the processor executes a computer subroutine stored in memory, it can perform the following steps: receiving a specified data table to be solidified via a human-computer interaction device.

[0158] For a description of the cache optimization management device provided in this embodiment of the invention, please refer to the aforementioned embodiments of the cache optimization management method; the embodiments of the present invention will not be repeated here.

[0159] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cache optimization management method as described in the foregoing embodiments.

[0160] For a description of the computer-readable storage medium provided in the embodiments of the present invention, please refer to the foregoing embodiments of the cache optimization management method; the embodiments of the present invention will not be repeated here.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section. It should also be noted that in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0162] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cache optimization management method, characterized by, The method comprises the following steps: counting the execution times of each data access task and the accessed hotness values of each data table in the disk within a preset time period based on the current time; taking the data table accessed by the data access task with the execution time greater than a first preset threshold as a to-be-solidified data table; taking the data table with the accessed hotness value greater than a second preset threshold as a to-be-solidified data table; persistently storing each to-be-solidified data table in a first memory pool so as to directly access each to-be-solidified data table from the first memory pool; wherein the accessed hotness value of each data table is positively correlated with the access times of the data table.

2. The cache optimization management method of claim 1, wherein, After counting the execution times of each data access task and the accessed hotness values of each data table in the disk within a preset time period based on the current time, the cache optimization management method further comprises the following steps: determining the hotness level to which the accessed hotness value of each data table belongs according to a preset hotness level interval; after taking the data table with the accessed hotness value greater than a second preset threshold as a to-be-solidified data table, the cache optimization management method further comprises the following steps: dividing an extra-capacity memory space from the current remaining capacity of the first memory pool; persistently storing each data table in the extra-capacity memory space in the order from high to low according to the hotness level.

3. The cache optimization management method of claim 2, wherein, After counting the execution times of each data access task and the accessed hotness values of each data table in the disk within a preset time period based on the current time, the cache optimization management method further comprises the following steps: persistently storing the calculation result data generated by the execution in the first memory pool after the data access task with the execution time greater than a first preset threshold is executed for the first time.

4. The cache optimization management method of claim 3, wherein, After counting the execution times of each data access task and the accessed hotness values of each data table in the disk within a preset time period based on the current time, the cache optimization management method further comprises the following steps: labeling the data table with the accessed hotness value lower than a third preset threshold with a removal tag; judging whether the data table persistently stored in the first memory pool and accessed has the removal tag; if yes, removing the data table from the first memory pool after the data table persistently stored in the first memory pool is used up.

5. The cache optimization management method of claim 4, wherein, further comprising a second memory pool; the cache optimization management method further comprises the following steps: every other preset period, judging whether the remaining capacity of the first memory pool is less than a fourth preset threshold; if yes, selecting a to-be-migrated task from the to-be-executed tasks of the first memory pool; adding the to-be-migrated task to the task queue of the second memory pool so that the second memory pool loads the data corresponding to the to-be-migrated task.

6. The cache optimization management method of claim 5, wherein, the cache optimization management method further comprises the following steps: judging whether the memory capacity required by the next to-be-executed data access task in the task queue exceeds the remaining capacity of the memory pool corresponding to the to-be-executed data access task; if yes, pausing the task queue in which the to-be-executed data access task is located; When it is monitored that the remaining capacity of the memory pool corresponding to the data access task to be executed meets the required memory capacity of the data access task to be executed, the task in the task queue is continued to be executed.

7. The cache optimization management method of any of claims 1 to 6, wherein, Before the each to-be-solidified data table is persisted in the first memory pool for direct access of each to-be-solidified data table from the first memory pool, the cache optimization management method further comprises: receiving the specified to-be-solidified data table through a human-computer interaction device.

8. A cache optimization management apparatus, characterized by, comprising: a statistical module configured to count the execution times of each data access task and the accessed hotness values of each data table in a disk within a past preset time length based on a current time; a first analysis module configured to take the data table accessed by the data access task with an execution time greater than a first preset threshold as a to-be-solidified data table; a second analysis module configured to take the data table with an accessed hotness value greater than a second preset threshold as a to-be-solidified data table; an execution module configured to persist each to-be-solidified data table in a first memory pool for direct access of each to-be-solidified data table from the first memory pool; wherein the accessed hotness value of each data table is positively correlated with the access times of the data table.

9. A cache optimization management device, characterized by, comprising: a memory configured to store a computer program; a processor configured to implement the steps of the cache optimization management method according to any one of claims 1 to 7 when the computer program is executed.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the cache optimization management method according to any one of claims 1 to 7.

Images