Cache processing method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a cache processing method, device and equipment and storage medium. The method comprises: receiving a data read request, the data read request being used to request reading target data; if an index of the target data corresponding to the data read request does not exist in a data index set, reading the target data from a first storage device and performing first data processing on the target data, the data index set being stored in a second storage device. The method is used to reduce the occupancy of memory resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for caching. Background Technology

[0002] Solid State Drive (SSD) is a type of hard drive based on a flash memory chip array. SSD performance and price fall between those of mechanical hard drives and RAM. In order to obtain better storage performance at a lower cost, storage systems typically use mechanical hard drives as the backend to obtain larger storage capacity, and use SSDs as a cache to accelerate read and write operations.

[0003] In related technologies, an SSD can be used as a preset hard drive, storing multiple data items and defining metadata information for each item. This metadata information can include data popularity values ​​and indexes. Popular data can be stored on the preset hard drive based on popularity values, while cold data can be evicted. The corresponding cached items can be retrieved from the preset hard drive using the index. This metadata information can be stored in memory. However, with the rapid increase in preset hard drive capacity, the memory resources consumed by metadata information are increasing, resulting in high memory resource utilization. Summary of the Invention

[0004] This application provides a caching processing method, apparatus, device, and storage medium to reduce the occupancy rate of memory resources.

[0005] In a first aspect, embodiments of this application provide a caching processing method, including:

[0006] Receive a data read request, the data read request being used to request the read of target data;

[0007] If the data index set does not contain an index for the target data corresponding to the data read request, the target data is read from the first storage device and the target data is subjected to first data processing. The data index set is stored in the second storage device.

[0008] In one possible implementation, the target data undergoes a first data processing step, including:

[0009] The heat value of the target data is updated to obtain a first heat set, which includes the heat values ​​of each data in a preset hard disk.

[0010] The target data is written to the preset hard disk, and the data index set is updated according to the current index of the target data.

[0011] In one possible implementation, the heat value of the target data is updated to obtain the first heat set, including:

[0012] The heat value of the target data is stored in the current heat set to obtain the intermediate heat set, and the current cache ratio of the preset hard disk is determined based on the intermediate heat set.

[0013] If the current cache ratio is greater than or equal to the second preset ratio, then the data of the preset hard disk is downgraded in terms of popularity until the cache ratio of the hot data of the preset hard disk is less than or equal to the second preset ratio, and the first popularity set is obtained.

[0014] The heat data refers to data whose heat value is greater than or equal to a preset heat value.

[0015] In one possible implementation, the preset hard disk includes multiple data blocks, each data block including multiple data items; hot data on the preset hard disk is downgraded until the current cache ratio of hot data on the preset hard disk is less than or equal to a second preset ratio, including:

[0016] Determine the corresponding current data block in the preset hard disk;

[0017] The current data block is subjected to a hotness degradation process to reduce the amount of hot data on the preset hard drive;

[0018] If the current cache ratio of hot data on the preset hard disk is greater than the second preset ratio, then the next data block in the preset hard disk will be downgraded in terms of hotness until the cache ratio of the preset hard disk is less than or equal to the second preset ratio.

[0019] In one possible implementation, determining the current cache ratio of the preset hard disk based on the intermediate heat set includes:

[0020] The first quantity of each data in the intermediate heat set is determined as the current cache quantity of hot data in the preset hard disk;

[0021] Obtain the total amount of thermal data, which is used to indicate the total quantity of thermal data;

[0022] The ratio of the current cache size to the total amount of hot data is determined as the current cache ratio.

[0023] In one possible implementation, writing the target data to the preset hard disk includes:

[0024] The target data is stored in a memory buffer;

[0025] If the amount of data in the memory buffer is greater than or equal to a first preset amount, then the data in the memory buffer is stored in the second data block of the preset hard disk, and the second data block is a newly added data block at the tail of the data block queue of the preset hard disk.

[0026] In one possible implementation, storing the data from the memory buffer to a second data block of the preset hard disk includes:

[0027] If the number of data blocks in the preset hard disk is less than the second preset amount, then the data in the memory buffer is stored in the second data block of the preset hard disk;

[0028] If the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, then the first data block of the preset hard disk is recycled, and the data in the memory buffer is stored in the second data block of the preset hard disk. The first data block is the earliest data block stored in the data block queue to the preset hard disk.

[0029] In one possible implementation, the first data block of the preset hard disk is reclaimed, including:

[0030] Multiple data points with indexes in the first data block are identified as multiple intermediate data points;

[0031] Based on the second heat set, determine the hot data among the plurality of intermediate data;

[0032] The hot data from the plurality of intermediate data is stored in a memory buffer, and the data index set is updated.

[0033] In one possible implementation, the first heat set includes multiple bitmaps, wherein each bitmap corresponds to heat values ​​of multiple data in a data block, the heat values ​​being indicated by binary multi-digit numbers.

[0034] In one possible implementation, the hot value of each data item is determined based on an improved cache hot management (MBC) algorithm, wherein the MBC algorithm updates the hot value of the data based on either a doubly linked Least Recently Used (LRU) algorithm or a Least Frequently Used (LFU) algorithm.

[0035] In one possible implementation, before the index of the target data corresponding to the data read request does not exist in the data index set, the method further includes:

[0036] If the first index set in memory contains an index of the target data, then the data index set contains an index of the target data, and the number of indexes in the first index set is less than the number of indexes in the data index set;

[0037] If the first index set in memory does not contain an index for the target data, then it is determined whether an index for the target data exists in the data index set.

[0038] In one possible implementation, the method further includes:

[0039] Receive a data deletion request and delete the target data corresponding to the data deletion request in the data index set.

[0040] In one possible implementation, the method further includes:

[0041] Receive a data update request and update the target data corresponding to the data update request in the data index set.

[0042] Secondly, embodiments of this application provide a cache processing apparatus, including a receiving module, a judging module, and a first data processing module:

[0043] The receiving module is used to receive a data read request, the data read request being used to request the reading of target data;

[0044] The reading module is used to read the target data from the first storage device if there is no index for the target data corresponding to the data reading request in the data index set.

[0045] The first data processing module is used to perform first data processing on the target data.

[0046] In one possible implementation, the first data processing module is specifically used for:

[0047] The heat value of the target data is updated to obtain a first heat set, which includes the heat values ​​of each data in a preset hard disk.

[0048] The target data is written to the preset hard disk, and the data index set is updated according to the current index of the target data.

[0049] In one possible implementation, the first data processing module is specifically used for:

[0050] The heat value of the target data is stored in the current heat set to obtain the intermediate heat set, and the current cache ratio of the preset hard disk is determined based on the intermediate heat set.

[0051] If the current cache ratio is greater than or equal to the second preset ratio, then the data of each data in the preset hard disk is downgraded in terms of popularity until the cache ratio of the hot data in the preset hard disk is less than or equal to the second preset ratio, and the first popularity set is obtained.

[0052] Among them, heat data refers to data whose heat value is greater than or equal to the preset heat value.

[0053] In one possible implementation, the preset hard disk includes multiple data blocks, each data block including multiple data; the first data processing module 13 is specifically used for:

[0054] Determine the current data block in the preset hard drive;

[0055] Perform hotness degradation on the current data block to reduce the amount of hot data on the preset hard drive;

[0056] If the current cache ratio of hot data on the preset hard drive is greater than the second preset ratio, then the next data block in the preset hard drive will be downgraded in terms of hotness until the cache ratio of the preset hard drive is less than or equal to the second preset ratio.

[0057] In one possible implementation, the first data processing module is specifically used for:

[0058] The first quantity of each data in the intermediate heat set is determined as the current cache quantity of hot data in the preset hard disk;

[0059] Obtain the total amount of heat data; the total amount of heat data is used to indicate the total quantity of heat data.

[0060] The current cache ratio is determined by the ratio of the current cache size to the total amount of hot data.

[0061] In one possible implementation, the first data processing module is used for:

[0062] Store the target data in a memory buffer;

[0063] If the amount of data in the memory buffer is greater than or equal to the first preset amount, the data in the memory buffer will be stored in the second data block of the preset hard disk. The second data block is a newly added data block at the tail of the data block queue of the preset hard disk.

[0064] In one possible implementation, the first data processing module is specifically used for:

[0065] If the number of data blocks in the preset hard disk is less than the second preset number, then the data in the memory buffer will be stored in the second data block of the preset hard disk.

[0066] If the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, the first data block of the preset hard disk is reclaimed, and the data in the memory buffer is stored in the second data block of the preset hard disk. The first data block is the earliest data block stored in the data block queue to the preset hard disk.

[0067] In one possible implementation, the first data processing module is specifically used for:

[0068] Multiple data items with indexes in the first data block are identified as multiple intermediate data items;

[0069] Based on the second set of heat data, identify the hot data among multiple intermediate data sets;

[0070] Store hot data from multiple intermediate data sets into a memory buffer.

[0071] In one possible implementation, the first heat set includes multiple bitmaps, wherein each bitmap corresponds to heat values ​​of multiple data in a data block, and the heat values ​​are indicated by binary multi-digit numbers.

[0072] In one possible implementation, the hot value of each data is determined based on the improved cache hot management (MBC) algorithm, wherein the MBC algorithm updates the hot value of the data based on either the least recently used (LRU) algorithm or the least frequently used (LFU) algorithm in a doubly linked list.

[0073] In one possible implementation, the device is further used for:

[0074] If the first index set in memory contains an index of the target data, then the data index set contains an index of the target data, and the number of indexes in the first index set is less than the number of indexes in the data index set;

[0075] If the first index set in memory does not contain an index for the target data, then it is determined whether an index for the target data exists in the data index set.

[0076] In one possible implementation, the device further includes:

[0077] Receive a data deletion request and delete the target data corresponding to the data deletion request in the data index set.

[0078] In one possible implementation, the device further includes:

[0079] Receive a data update request and update the target data corresponding to the data update request in the data index set.

[0080] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0081] The memory stores computer-executed instructions;

[0082] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0083] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0084] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0085] The caching processing method, apparatus, device, and storage medium provided in this application embodiment can store the indexes and popularity values ​​of each data in a preset hard disk separately, and store the data index set corresponding to the indexes of each data in the preset hard disk in a second storage device, which can reduce the memory resource occupancy rate. Attached Figure Description

[0086] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0087] Figure 1 A schematic diagram illustrating the application scenarios provided in this application;

[0088] Figure 2 A flowchart illustrating the caching method provided in this application;

[0089] Figure 3 A schematic diagram of an architecture for updating heat values ​​is provided in an embodiment of this application;

[0090] Figure 4 A schematic diagram of a reading process architecture provided for an embodiment of this application;

[0091] Figure 5 A schematic diagram illustrating a data popularity degradation process provided in an embodiment of this application;

[0092] Figure 6 A schematic diagram of an architecture for data popularity degradation provided in an embodiment of this application;

[0093] Figure 7 A schematic diagram of a data writing process provided in an embodiment of this application;

[0094] Figure 8 This application provides a schematic diagram of a data writing architecture.

[0095] Figure 9 A schematic diagram of a recycling process architecture provided in an embodiment of this application;

[0096] Figure 10 A schematic diagram of a data block of a preset hard disk provided in an embodiment of this application;

[0097] Figure 11 A schematic diagram of a caching processing method provided in an embodiment of this application;

[0098] Figure 12 This application provides a schematic diagram of the structure of a cache processing device according to an embodiment of the present application;

[0099] Figure 13 This is a schematic diagram of the structure of an electronic device provided in this application.

[0100] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0101] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0102] Figure 1 A schematic diagram illustrating the application scenarios provided in this application. Please refer to [link / reference]. Figure 1 The system includes an electronic device 101, a first storage device 102, and a second storage device 103. The electronic device 101 may include a preset hard disk and memory. The preset hard disk may be a non-volatile memory, meaning data is retained even after power is off. The preset hard disk stores data using flash memory chips, allowing for fast read and write speeds. The preset hard disk can store frequently accessed data; for example, it may be a high-speed storage device such as an SSD or general-purpose flash memory. A data index set stores indexes of data within the preset hard disk. This data index set may be stored in the second storage device 103. The first popularity set may include popularity values ​​for each piece of data within the preset hard disk.

[0103] Electronic device 101 can receive data read requests. It can determine whether an index corresponding to the target data for the data read request exists in the data index set. If not, the target data can be read from the first storage device 102, and the target data can be processed in the first data processing.

[0104] Electronic device 101 can be a server, laptop, mobile phone, etc., and first storage device 102 can be a mechanical hard drive.

[0105] In related technologies, a preset hard drive can store multiple data sets, and metadata information for these data sets can be determined. This metadata information can include data popularity values ​​and indexes. Popular data can be stored on the preset hard drive based on popularity values, while cold data can be discarded. The corresponding target data can be retrieved from the preset hard drive using the index. This metadata information can be stored in memory. However, with the rapid increase in preset hard drive capacity, the memory resources consumed by metadata information are increasing, resulting in high memory resource utilization.

[0106] The caching method provided in this application embodiment can store the index and popularity value of each data in the preset hard disk separately, and store the data index set of each data in the preset hard disk in the second storage device, which can reduce the memory resource usage rate.

[0107] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0108] Figure 2 A flowchart illustrating the caching method provided in this application. Please refer to [link / reference]. Figure 2 The method may include:

[0109] S201, Receive data read request.

[0110] The execution entity in this application embodiment can be a server or a cache processing device set in the server. The cache processing device can be implemented by software or by a combination of software and hardware.

[0111] The target data can be stored in the first storage device. In order to improve the data read and write speed, hot data can be stored in the preset hard disk, which can be a solid-state drive.

[0112] The preset hard drive can be divided into multiple data chunks using the First-In-First-Out (FIFO) SSD cache (FSC) algorithm. Each data chunk can contain multiple data items, and the data chunk queue can be a First-In-First-Out (FIFO) queue.

[0113] A data read request can be used to request the read of target data.

[0114] S202. If there is no index for the target data corresponding to the data read request in the data index set, then read the target data from the first storage device.

[0115] Hot data can be stored in a preset hard disk. All data can be stored in the first storage device. Hot data can be data with a heat value greater than or equal to a preset heat value. If the heat value of hot data in the preset hard disk is less than the preset heat value, the data will change from hot data to cold data. Cold data can be deleted from the preset hard disk.

[0116] The number of indexes in the first index set is less than the number of indexes in the data index set.

[0117] The first index set may include some indexes from the data index set, and the data index set may include indexes of all data in the preset hard disk. The first index set may be cached in memory, and the data index set may be stored in a second storage device. The second storage device may be organized in the form of a database. For example, the preset database may be the Rocks database in a KV (Key-Value) database. In this case, the first index set may be served by the database's built-in cache.

[0118] For example, the default database is a KV database, where the Key is used to indicate the data identifier and the Value is used to indicate the storage location of the data in the default database.

[0119] After hot data is written to the preset hard drive, since the data index will not change before the data is evicted from the preset hard drive, the data index can be saved to the second storage device. That is, storing the data index set to the second storage device can reduce memory consumption.

[0120] Furthermore, if the first index set in memory contains an index of the target data, then the data index set contains an index of the target data; if the first index set in memory does not contain an index of the target data, then it is determined whether an index of the target data exists in the data index set.

[0121] If all the indexes of the data are stored in the second storage device through the data index set, the index needs to be read from the second storage device every time the target data is read. Therefore, storing part of the index in memory through the first index set can improve the efficiency of obtaining the target data.

[0122] It is worth noting that the index ratio can be used to determine the number of indexes in the first index set. The index ratio can be used to indicate the ratio of the cache amount in the first index set to the total number of indexes in the data index set. The larger the index ratio, the more indexes there are in the first index set, the higher the data reading speed efficiency, but the more memory is consumed when reading data.

[0123] The index ratio can range from 0 to 100%, for example, the index ratio can be 20%, or the index ratio can be determined by the size of the memory.

[0124] If the data index set contains an index of the target data corresponding to the data read request, the target data can be read according to the index of the target data, and the popularity value of the target data can be updated. The update of the popularity value of the target data can be found in the following example, which will not be repeated here.

[0125] S203. Perform the first data processing on the target data.

[0126] In some possible embodiments, the popularity value of the target data can be updated to obtain a first popularity set; the target data can be written to a preset hard disk, and the data index set can be updated according to the current index of the target data.

[0127] If the target data is hot data, it can be stored on a preset hard disk, and the index of the target data can be the index on the preset hard disk; in addition, since a buffer is set in memory, the index of the target data can also be the index in memory.

[0128] The first set of data popularity can include the popularity values ​​of each data in the preset hard drive. The smallest unit of data reading is a line, that is, each data is read in lines.

[0129] The heat value of the target data can be updated in the following way to obtain the first heat set: store the heat value of the target data into the current heat set to obtain the intermediate heat set, and determine the current cache ratio of the preset hard disk based on the intermediate heat set; if the current cache ratio is greater than or equal to the second preset ratio, perform heat downgrading processing on each data of the preset hard disk until the cache ratio of the hot data of the preset hard disk is less than or equal to the second preset ratio to obtain the first heat set.

[0130] The current popularity set can be the popularity set stored in memory before the popularity value of the target data is stored.

[0131] The intermediate heat set can be the transitional heat set after the heat value of the target data is stored in memory. At this time, compared with before the update, the heat value of the target data has been added to the heat set in memory. It is necessary to determine the current cache ratio of the intermediate heat set. If the current cache ratio is less than the second preset ratio, the intermediate heat set is determined as the first heat set.

[0132] The more hot data there is, the more data is inserted back into the buffer from the data block, which can improve the efficiency of writing data to fill the buffer, and thus improve the efficiency of writing data to the preset hard drive. By adjusting the amount of hot data determined in the FSC algorithm through the current cache ratio, the amount of data written to the preset hard drive can be controlled, which can extend the life of the SSD.

[0133] Furthermore, the current cache ratio can be determined as follows: the first number of the heat values ​​of each data in the intermediate heat set is determined as the current cache quantity of hot data in the preset hard disk; the total amount of hot data is obtained, and the ratio of the current cache quantity to the total amount of hot data is determined as the current cache ratio.

[0134] The total amount of heat data can be used to indicate the total quantity of heat data.

[0135] Hot data refers to data with a heat value greater than or equal to the preset heat value, while cold data has a heat value less than the preset heat value.

[0136] For any given piece of data, its popularity value needs to be modified actively or passively each time the data is accessed. The popularity values ​​of each piece of data can be stored in the first popularity set in memory to ensure the efficiency of data reading and writing.

[0137] An improved Multi-Bit CLOCK (MBC) algorithm can be used to determine the popularity value of each piece of data, which can then be stored in memory. In one embodiment, data that needs to be written to a preset hard disk is stored in a buffer in memory. Once the cache size of each piece of data in the buffer reaches a preset cache size, the data in the buffer can be stored in the preset hard disk.

[0138] Small IO writes allow data to be temporarily stored in a memory cache for writing, deleting, and modifying. Only after a certain amount of data has accumulated are they written in blocks to the SSD, which can extend the SSD's lifespan. In small IO writes, "IO" stands for Input / Output. Small IO writes refer to write operations involving relatively small data blocks. In some implementations, IO operations smaller than 16KB are considered "small" IO writes.

[0139] The improved MBC algorithm updates the heat value based on the Least Recently Used (LRU) algorithm or the Least Frequently Used (LFU) algorithm of a doubly linked list. The heat value of each data can be indicated by a bitmap. The first heat set can include multiple bitmaps, where each bitmap corresponds to a heat value of a data block.

[0140] Popularity values ​​can be indicated using binary multi-digit numbers, such as 2, 4, 8, etc. The higher the number of digits, the more popularity information can be retained, but the more memory is required.

[0141] For example, a 2-bit flag can be used to indicate the popularity value of the data. The popularity value can be 00, 01, 10, or 11. A popularity value of 00 can indicate that the data is in a cold state and can be discarded, while popularity values ​​of 01, 10, and 11 can indicate that the data is in a hot state and needs to be retained, that is, the data is hot data.

[0142] By mapping the location of data in a data block to its location in a bitmap, the indexing of heat values ​​in the bitmap can be reduced, improving the efficiency of determining data heat values ​​and thus improving the efficiency of data processing.

[0143] In addition, bitmaps consume memory at the bit level, while algorithms such as LRU consume memory at the byte level. By using bitmaps to record the popularity value of data, memory consumption can be reduced.

[0144] The MBC algorithm based on the LFU algorithm can increment the popularity of the target data by 1 after each reading. For example, if the popularity value of the target data is 00, the updated popularity value will be 01; if the popularity value of the target data is 01, the updated popularity value will be 10.

[0145] The MBC algorithm based on the LRU algorithm can increase the popularity value of the target data to the highest level after each reading. For example, if the popularity value of the target data is 00, the updated popularity value is 11; if the popularity value of the target data is 10, the updated popularity value is 11.

[0146] The following describes the process of updating the popularity value provided in this application embodiment, using the LRU algorithm with a 2-bit flag as an example.

[0147] Figure 3 This is a schematic diagram of an architecture for updating heat values ​​provided in an embodiment of this application. Please refer to [link / reference]. Figure 3For any given target data, assuming the highest popularity value is 11 and the lowest popularity value is 00, if the current popularity value of the target data is 00, and the target data corresponding to the data read request has been identified, the target data has been matched. The LRU algorithm can then be used to determine the current popularity value of the target data as 11. For example, if the current popularity value of the target data is 01, 10, or 11, and the target data is stored on a preset hard drive, after the data read request reads the target data, the target data has been matched, and its current popularity value is directly determined as 11.

[0148] In addition, since the popularity of each data hit is upgraded, the number of hot data will increase. When downgrading the popularity value of all data, the popularity value of the data can be reduced by one level. If the current popularity value is 00, it will remain at 00; if the current popularity value is 01, it will be reduced to 00; if the current popularity value is 10, it will be reduced to 01; if the current popularity value is 11, it will be reduced to 10.

[0149] Before writing the target data to the preset hard disk, the target data can be written to the buffer. After the buffer reaches the preset buffer size, the data in the buffer is stored in the preset hard disk, including the target data, thus realizing the writing of the target data to the preset hard disk.

[0150] After writing the target data to the preset hard disk, the index of the target data can be updated to obtain the second index set.

[0151] Since a first preset ratio for caching indexes in memory is set, after updating the index of the target data to the first index set, the number of cached indexes in the second index set increases. The current index ratio in memory can be determined based on the number of indexes in the second index set. If the current index ratio is greater than the first preset ratio, the indexes in the second index set in memory are deleted to obtain the third index set in memory.

[0152] It's worth noting that deleting an index in memory does not affect the data index set stored on the second storage device. The data index set is deleted and updated based on data deletion or update requests.

[0153] In some possible implementations, if a data deletion request is received, the target data can be deleted from the data index set.

[0154] After removing the index of the target data from the data index set, the target data cannot be retrieved from the solid-state drive (SSD), thus enabling the deletion of the target data from the SSD.

[0155] After deleting the index of the target data, if the index of the target data exists in the heat set in memory, the heat value of the target data in the heat set can be deleted directly. Alternatively, after deleting the index of the target data, exit directly. When recycling the data in the preset hard disk, the target data and the corresponding heat value will be discarded because the index cannot be found. The heat value corresponding to the target data will be deleted in memory, and the target data will be deleted in the preset hard disk.

[0156] In some possible implementations, if a data update request is received, the target data can be updated in the data index set.

[0157] Data update requests can be used to update the index of target data. For example, assuming the index of the target data is index 1, after the target data is updated and then re-stored to the preset hard disk, the index of the target data can be updated to the index 2 corresponding to the new location.

[0158] After updating the index of the target data corresponding to the data update request from the data index set, the target data cannot be retrieved from the solid-state drive, thus enabling the deletion of the target data from the solid-state drive.

[0159] It is worth noting that when updating target data, the target data can be written to a buffer before being written to the preset hard drive.

[0160] The caching method provided in this application embodiment can store the indexes and popularity values ​​of each data in a preset hard disk separately, store the data index set of all indexes in a second storage device, and store the first popularity set in memory, which can reduce the memory resource usage. At the same time, storing the popularity values ​​of each data and the first index set of some indexes in memory can ensure the efficiency of caching processing.

[0161] Based on the above embodiments, the following is combined with Figure 4 The data reading process provided in the embodiments of this application will be further explained.

[0162] Figure 4 This is a schematic diagram illustrating the architecture of a reading process provided in an embodiment of this application. Please refer to [link / reference]. Figure 4After receiving a data read request, it can determine whether the target data index exists in the first index set. If so, the target data can be read from the preset hard disk and the popularity value of the target data can be updated. If not, it can determine whether the target data index exists in the data index set. If so, the target data can be read from the preset hard disk and the popularity value of the target data can be updated. If not, the target data can be read from the first storage device and the popularity value of the target data can be updated. It can then determine whether to write the target data to the preset hard disk. If so, the target data can be written to the preset hard disk.

[0163] It can determine whether the current cache ratio of the preset hard disk is greater than or equal to the second preset ratio. If so, it will perform an eviction process on the data blocks in the preset hard disk.

[0164] Based on the above embodiments, the following is combined with Figure 5 and Figure 6 The execution process of downgrading the popularity of the data provided in the embodiments of this application will be described.

[0165] Figure 5 This is a schematic diagram illustrating a data popularity degradation process provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 The method may include:

[0166] S501. Determine the current data block in the preset hard disk.

[0167] After the last data block was evicted, the pointer can be moved to the next data block after the data block that was evicted. The next data block is the current data block, and the current data popularity degradation can start from the current data block.

[0168] The next data block can be a data block that entered the data block queue earlier than the previous data block.

[0169] For example, suppose there are 5 data blocks in the preset hard drive, namely data blocks 1-5. Suppose the data block queue is: data block 1, data block 2, data block 3, data block 4 and data block 5. Suppose data block 1 enters the data block queue first and data block 5 enters the data block queue last. Suppose that when the first data is evicted, the last data block is data block 4, then the pointer can be moved to data block 3. When the second data is evicted, the current data block is data block 3.

[0170] S502. Perform hotness degradation processing on the current data block to reduce the amount of hot data on the preset hard drive.

[0171] The current data block can include multiple data points, and multiple data points can be subject to popularity degradation processing. The popularity degradation process can be found in [link to relevant documentation]. Figure 3 The execution process within.

[0172] S503. If the current cache ratio of hot data on the preset hard disk is greater than the second preset ratio, then the next data block in the preset hard disk is subjected to hot downgrading processing until the cache ratio of the preset hard disk is less than or equal to the second preset ratio.

[0173] If the current cache ratio of hot data on the preset hard drive is less than or equal to the second preset ratio, then the degradation stops and the pointer moves to the next data block.

[0174] The process for determining the current cache ratio can be found in S204, and will not be repeated here.

[0175] The process of recycling data from the preset hard drive can be found in the following example, and will not be repeated here.

[0176] Figure 6 This is a schematic diagram of an architecture for data hot-count degradation provided in an embodiment of this application. Please refer to [link / reference]. Figure 6 When there is too much hot data, where the current cache ratio of hot data on the preset hard drive is greater than the second preset ratio, the current data block corresponding to the pointer can be determined. The data corresponding to the current data block is downgraded by the improved MBC algorithm until the current cache ratio of hot data on the preset hard drive is less than or equal to the second preset ratio, and then the pointer is pointed to the next data block.

[0177] The caching processing method provided in this application embodiment can downgrade the hotness of each data on the preset hard disk when the current cache ratio is greater than or equal to the second preset ratio, until the current cache ratio of the hot data on the preset hard disk is less than or equal to the second preset ratio. This can reduce the amount of hot data on the preset hard disk and reduce memory consumption.

[0178] Based on the above embodiments, the following, in conjunction with Figure 7 and Figure 8 The execution process of writing target data to a preset hard disk according to the embodiments of this application will be described.

[0179] Figure 7 This is a schematic diagram illustrating a data writing process provided in an embodiment of this application. Please refer to... Figure 7 The method may include:

[0180] S701, Store the target data in a memory buffer.

[0181] After storing the target data in a memory buffer, the index of the target data can be stored in the first index set and the data index set.

[0182] The buffer can include newly written target data as well as hot data recycled from the preset hard disk.

[0183] S702. If the amount of data in the memory buffer is greater than or equal to the first preset amount, then the data in the memory buffer is stored in the second data block of the preset hard disk.

[0184] The initial size of the buffer can be the same as the size of a data block in the default hard disk.

[0185] The second data block can be a newly added data block in the data block queue of the preset hard disk.

[0186] Furthermore, if the number of data blocks in the preset hard disk is less than the second preset amount, the data in the memory buffer is stored in the second data block of the preset hard disk; if the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, the first data block of the preset hard disk is reclaimed, and the data in the memory buffer is stored in the second data block of the preset hard disk.

[0187] The first data block can be the earliest data block in the data block queue to be stored on the preset hard drive.

[0188] The second preset value is used to indicate the maximum number of data blocks stored in the preset hard disk.

[0189] The data block queue includes a second preset number of data blocks. If there are free data blocks in the data block queue, they can be stored directly in the buffer without being recycled. If there are no free data blocks in the data block queue, recycling is performed to increase the number of free data blocks.

[0190] Figure 8 This is a schematic diagram of a data writing architecture provided in an embodiment of this application. Please refer to... Figure 8 The target data can be stored in a memory buffer. It is determined whether the amount of data in the memory buffer is less than a first preset amount. If so, the process ends. If not, the data in the memory buffer is stored in a second data block on a preset hard disk. It is determined whether the number of data blocks in the preset hard disk is greater than the second preset amount. If so, the data in the preset hard disk can be recycled. The popularity can be updated, and the cache ratio of hot data in the preset hard disk can be determined. If there is too much hot data, a data block eviction process is performed.

[0191] Furthermore, multiple data points with indexes in the first data block can be identified as multiple intermediate data points. Based on the second heat set, hot data points among the multiple intermediate data points can be identified. The hot data points among the multiple intermediate data points can be stored in a memory buffer, and the data index set can be updated.

[0192] During the data reclamation process, valid data, particularly "hot" data, can be reclaimed from data blocks. Valid data refers to data that exists in the data index set and has not been deleted or modified.

[0193] The hot data among multiple intermediate data can be identified through the second set of heat data, and the hot data among multiple intermediate data can be stored in a memory buffer.

[0194] When updating the data index set, you can delete indexes of data that has not been recycled, and update indexes of data that has been recycled.

[0195] It is worth noting that after executing the data deletion and modification process, the data in the preset hard drive is not deleted. Only the index of the data in the data index set is deleted, which can be regarded as the data being deleted. The data is only deleted from the preset hard drive when the data of each data is recycled. This can reduce the update frequency of the preset hard drive and improve the lifespan of the preset hard drive.

[0196] Figure 9 This is a schematic diagram of a recycling process architecture provided in an embodiment of this application. Please refer to [link / reference]. Figure 9 If the number of data blocks in the preset hard disk is greater than the second preset number, the first data block at the head of the data block queue can be deleted, and the data that is valid and hot data in the deleted first data block can be stored in the memory buffer to retain the hot data.

[0197] After hot data is stored in the buffer to a preset size, each data in the buffer can be stored at the end of the data block queue on the preset hard disk to form a second data block.

[0198] The caching method provided in this application embodiment can store data in a memory buffer. After the amount of data in the buffer reaches a first preset amount, the data in the buffer is stored in a preset hard disk. The deleted data is only deleted from the preset hard disk during the recycling process, which can reduce the updates of the preset hard disk and improve the service life of the preset hard disk.

[0199] Figure 10 This is a schematic diagram of a data block on a preset hard disk provided in an embodiment of this application. Please refer to [link / reference]. Figure 3The preset hard disk contains a data block queue with four data blocks, designated as data blocks 1-4, where the first data block is data block 1. Hit data can be inserted into the memory cache. Once the data size in the memory buffer is greater than or equal to a preset size (i.e., the buffer corresponding to the data block size is full), data can be stored in the preset hard disk buffer, resulting in the second data block, data block 5. If the preset second data block size on the preset hard disk is 4, then the first data block stored in the data block queue, data block 1, can be deleted. This implements a first-in-first-out (FIFO) data block system and partially reclaims the data in data block 1, returning the hot data in data block 1 to the buffer.

[0200] This feature allows configuring preset popularity values ​​for some of the recycled data, and data with popularity values ​​greater than the preset popularity value will be recycled. For example, if the preset popularity value is 00, then data with popularity values ​​of 01, 10, and 11 will be recycled; if the preset popularity value is 01, then data with popularity values ​​of 10 and 11 will be recycled.

[0201] Figure 11 This is a schematic diagram illustrating the structure of a caching processing method provided in an embodiment of this application. Please refer to [link / reference]. Figure 11 It can include a pre-installed hard drive, memory, and a secondary storage device. The pre-installed hard drive can include multiple data blocks, where the first data block is the head data block stored first in the data block queue, and the second data block is the tail data block stored last in the data block queue.

[0202] The first heat set stores the heat values ​​of each data item in the preset hard disk in memory. This first heat set can include multiple bitmaps, each with its corresponding data block, and the heat value of each data item within each data block is included in the bitmap. A data index set for all data can be stored in the second storage device. A partial first index set for the data is stored in memory.

[0203] The memory buffer can store hot data that is read from the first storage device, valid hot data in the first data block deleted from the preset hard disk, and newly written data. When the data in the buffer reaches a first preset amount, the data in the buffer can be stored in the data block queue of the preset hard disk, wherein the data in the buffer is stored as the tail data block of the data block queue.

[0204] When storing the data in the buffer to the preset hard disk, if the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, the first data block in the preset hard disk can be deleted, and the valid hot data in the first data block can be recycled to the buffer in memory according to the first heat set and the data index set.

[0205] Figure 12This is a schematic diagram of a caching processing apparatus provided in an embodiment of this application. Please refer to [link / reference]. Figure 12 The buffer processing device 10 may include a receiving module 11, a judging module 12, and a first data processing module 13.

[0206] The receiving module 11 is used to receive a data read request, which is used to request the reading of target data;

[0207] The reading module 12 is used to read the target data from the first storage device if the data index set does not contain an index for the target data corresponding to the data reading request;

[0208] The first data processing module 13 is used to perform first data processing on the target data.

[0209] The cache processing apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0210] In one possible implementation, the first data processing module 13 is specifically used for:

[0211] The popularity value of the target data is updated to obtain the first popularity set, which includes the popularity values ​​of each data in the preset hard drive;

[0212] Write the target data to the preset hard disk and update the data index set according to the current index of the target data.

[0213] In one possible implementation, the first data processing module 13 is specifically used for:

[0214] The heat value of the target data is stored in the current heat set to obtain the intermediate heat set, and the current cache ratio of the preset hard disk is determined based on the intermediate heat set.

[0215] If the current cache ratio is greater than or equal to the second preset ratio, then the data of each data in the preset hard disk is downgraded in terms of popularity until the cache ratio of the hot data in the preset hard disk is less than or equal to the second preset ratio, and the first popularity set is obtained.

[0216] Among them, heat data refers to data whose heat value is greater than or equal to the preset heat value.

[0217] In one possible implementation, the preset hard disk includes multiple data blocks, each data block including multiple data; the first data processing module 13 is specifically used for:

[0218] Determine the current data block in the preset hard drive;

[0219] Perform hotness degradation on the current data block to reduce the amount of hot data on the preset hard drive;

[0220] If the current cache ratio of hot data on the preset hard drive is greater than the second preset ratio, then the next data block in the preset hard drive will be downgraded in terms of hotness until the cache ratio of the preset hard drive is less than or equal to the second preset ratio.

[0221] In one possible implementation, the first data processing module 13 is specifically used for:

[0222] The first quantity of each data in the intermediate heat set is determined as the current cache quantity of hot data in the preset hard disk;

[0223] Obtain the total amount of heat data; the total amount of heat data is used to indicate the total quantity of heat data.

[0224] The current cache ratio is determined by the ratio of the current cache size to the total amount of hot data.

[0225] In one possible implementation, the first data processing module 13 is used for:

[0226] Store the target data in a memory buffer;

[0227] If the amount of data in the memory buffer is greater than or equal to the first preset amount, the data in the memory buffer will be stored in the second data block of the preset hard disk. The second data block is a newly added data block at the tail of the data block queue of the preset hard disk.

[0228] In one possible implementation, the first data processing module 13 is specifically used for:

[0229] If the number of data blocks in the preset hard disk is less than the second preset number, then the data in the memory buffer will be stored in the second data block of the preset hard disk.

[0230] If the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, the first data block of the preset hard disk is reclaimed, and the data in the memory buffer is stored in the second data block of the preset hard disk. The first data block is the earliest data block stored in the data block queue to the preset hard disk.

[0231] In one possible implementation, the first data processing module 13 is specifically used for:

[0232] Multiple data items with indexes in the first data block are identified as multiple intermediate data items;

[0233] Based on the second set of heat data, identify the hot data among multiple intermediate data sets;

[0234] Store hot data from multiple intermediate data sets into a memory buffer.

[0235] In one possible implementation, the first heat set includes multiple bitmaps, wherein each bitmap corresponds to heat values ​​of multiple data in a data block, and the heat values ​​are indicated by binary multi-digit numbers.

[0236] In one possible implementation, the hot value of each data is determined based on the improved cache hot management (MBC) algorithm, wherein the MBC algorithm updates the hot value of the data based on either the least recently used (LRU) algorithm or the least frequently used (LFU) algorithm in a doubly linked list.

[0237] In one possible implementation, the device is also used for:

[0238] If the first index set in memory contains an index of the target data, then the data index set also contains an index of the target data. The number of indexes in the first index set is less than the number of indexes in the data index set.

[0239] If the first index set in memory does not contain an index for the target data, then the existence of an index for the target data is checked in the data index set.

[0240] In one possible implementation, the device further includes:

[0241] Receive a data deletion request and delete the target data corresponding to the data deletion request from the data index set.

[0242] In one possible implementation, the device further includes:

[0243] Receive data update requests and update the target data corresponding to the data update request in the data index set.

[0244] The cache processing apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0245] Figure 13 A schematic diagram of the structure of an electronic device provided in this application. Please refer to [link / reference]. Figure 13 The electronic device 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0246] Memory 22 stores instructions executed by the computer;

[0247] The processor 21 executes computer execution instructions stored in the memory 22, causing the processor 21 to perform the cache processing method as described in the above method embodiment.

[0248] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the cache processing method of the above method embodiments.

[0249] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the cache processing method shown in the above method embodiments.

[0250] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0251] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0252] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0253] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0254] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0255] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0256] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0257] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0258] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for processing cache, characterized in that, Applied to electronic devices, including: Receive a data read request, the data read request being used to request the read of target data; If the data index set does not contain an index for the target data corresponding to the data read request, the target data is read from the first storage device and the target data is subjected to first data processing. The data index set is stored in the second storage device.

2. The method according to claim 1, characterized in that, The target data undergoes initial data processing, including: The heat value of the target data is updated to obtain a first heat set, which includes the heat values ​​of each data in a preset hard disk. The target data is written to the preset hard disk, and the data index set is updated according to the current index of the target data.

3. The method according to claim 2, characterized in that, The popularity value of the target data is updated to obtain the first popularity set, including: The heat value of the target data is stored in the current heat set to obtain the intermediate heat set, and the current cache ratio of the preset hard disk is determined based on the intermediate heat set. If the current cache ratio is greater than or equal to the second preset ratio, then the data of the preset hard disk is downgraded in terms of popularity until the cache ratio of the hot data of the preset hard disk is less than or equal to the second preset ratio, and the first popularity set is obtained. The heat data refers to data whose heat value is greater than or equal to a preset heat value.

4. The method according to claim 3, characterized in that, The preset hard disk includes multiple data blocks, and each data block includes multiple data items; the data on the preset hard disk is subjected to hotness degradation processing until the current cache ratio of the hot data on the preset hard disk is less than or equal to the second preset ratio, including: Determine the corresponding current data block in the preset hard disk; The current data block is subjected to a hotness degradation process to reduce the amount of hot data on the preset hard drive; If the current cache ratio of hot data on the preset hard disk is greater than the second preset ratio, then the next data block in the preset hard disk will be downgraded in terms of hotness until the cache ratio of the preset hard disk is less than or equal to the second preset ratio.

5. The method according to claim 3, characterized in that, Based on the intermediate heat set, the current cache ratio of the preset hard drive is determined, including: The first quantity of each data in the intermediate heat set is determined as the current cache quantity of hot data in the preset hard disk; Obtain the total amount of thermal data, which is used to indicate the total quantity of thermal data; The ratio of the current cache size to the total amount of hot data is determined as the current cache ratio.

6. The method according to claim 2, characterized in that, Writing the target data to the preset hard disk includes: The target data is stored in a memory buffer; If the amount of data in the memory buffer is greater than or equal to a first preset amount, then the data in the memory buffer is stored in the second data block of the preset hard disk, and the second data block is a newly added data block at the tail of the data block queue of the preset hard disk.

7. The method according to claim 6, characterized in that, Storing the data from the memory buffer into the second data block of the preset hard disk includes: If the number of data blocks in the preset hard disk is less than the second preset amount, then the data in the memory buffer is stored in the second data block of the preset hard disk; If the number of data blocks in the preset hard disk is greater than or equal to the second preset amount, then the first data block of the preset hard disk is recycled, and the data in the memory buffer is stored in the second data block of the preset hard disk. The first data block is the earliest data block stored in the data block queue to the preset hard disk.

8. The method according to claim 7, characterized in that, The process of reclaiming the first data block of the preset hard disk includes: Multiple data points with indexes in the first data block are identified as multiple intermediate data points; Based on the second heat set, determine the hot data among the plurality of intermediate data; The hot data from the plurality of intermediate data is stored in a memory buffer, and the data index set is updated.

9. The method according to any one of claims 1-8, characterized in that, The first heat set includes multiple bitmaps, wherein each bitmap corresponds to the heat values ​​of multiple data in a data block, and the heat values ​​are indicated by binary multi-digit numbers.

10. The method according to claim 9, characterized in that, The hot value of each data is determined based on the improved cache hot management MBC algorithm, wherein the MBC algorithm updates the hot value of the data based on the least recently used (LRU) algorithm or the least frequently used (LFU) algorithm of the doubly linked list.

11. The method according to any one of claims 1-10, characterized in that, Before the index of the target data corresponding to the data read request does not exist in the data index set, it also includes: If the first index set in memory contains an index of the target data, then the data index set contains an index of the target data, and the number of indexes in the first index set is less than the number of indexes in the data index set; If the first index set in memory does not contain an index for the target data, then it is determined whether an index for the target data exists in the data index set.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: Receive a data deletion request and delete the target data corresponding to the data deletion request in the data index set.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: Receive a data update request and update the target data corresponding to the data update request in the data index set.

14. A cache processing apparatus, characterized in that, Includes a receiving module, an acquisition module, and a first data processing module: The receiving module is used to receive data read requests, which are data read requests, data update requests, or data deletion requests. The reading module is used to read the target data from the first storage device if the data index set does not contain an index for the target data corresponding to the data reading request; The first data processing module is used to perform first data processing on the target data.

15. An electronic device, characterized in that, include: Memory and processor The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the cache processing method as described in any one of claims 1 to 13.

16. A computer program product comprising a computer program that, when executed by a processor, implements the cache processing method according to any one of claims 1 to 13.

Images