Data caching methods and apparatus, electronic devices and storage media
By configuring cache and backup cache in the database node, the persistent cache space is dynamically managed, which solves the problem of low data hit rate in the shared storage architecture and achieves high-efficiency data access and service performance.
Patent Information
- Application Number
- CN202510005965.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-02
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-01-02
AI Technical Summary
In a shared storage architecture, the persistent cache data hit rate of database nodes is low, especially when the space changes dynamically, and existing technologies cannot effectively improve data access efficiency.
By configuring cache and backup cache in the database node, the persistent cache space of the local storage device can be dynamically managed, data can be deleted or added to maintain the hit rate, and data that is frequently accessed or has high popularity can be prioritized.
It improved the data access hit rate of database nodes, reduced data service latency and improved efficiency, ensuring high-performance data services when space changes.
Smart Images

Figure CN119396888B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of database technology, and more particularly to a data caching method and apparatus, electronic device and storage medium. Background Technology
[0002] With the exponential growth of data volume in the computer field and the development of database technology, the shared storage method has gradually emerged. This method involves setting up a shared data layer and multiple database nodes that communicate with it. User data and internal system metadata can be stored in the shared data layer for multiple database nodes to share. The database nodes will persistently cache some data in the shared data layer locally and provide data services to the outside world accordingly.
[0003] Besides data caching, the local storage space of database nodes also needs to perform many other functions, and the data volume requirements of business operations on the database nodes also change. Therefore, the space allocated to persistent caching on the database node is dynamically changing. However, in related technologies, the implementation of persistent cache data on database nodes in scenarios with dynamically changing space is not ideal, and there is still room for improvement in the data hit rate of persistent cache. Summary of the Invention
[0004] In view of the above, one or more embodiments of this specification provide a data caching method and apparatus, electronic device and storage medium.
[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a data caching method is proposed, the method comprising: If the space occupied by the persistent cache data on the local storage device is greater than the persistent cache threshold, then some of the data in the persistent cache on the local storage device will be deleted so that the space occupied by the persistent cache data on the local storage device is not greater than the persistent cache threshold, and the metadata of the deleted data will be transferred from the memory cache area to the backup cache area. If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, then some or all of the metadata in the backup cache area in memory will be cached on the local storage device, and the metadata of the cached data will be transferred from the backup cache area in memory to the cache area. The persistent cache threshold is the space threshold allocated by the database node to the local storage device for persistent caching.
[0006] In one possible embodiment of this specification, the method further includes: If the space required by the data to be cached is greater than the remaining space in the space used for persistent caching on the local storage device, then some data in the persistent cache data on the local storage device is deleted, so that the data to be cached is cached in the local storage device, and the metadata of the deleted data is transferred from the cache area in memory to the backup cache area.
[0007] In one possible embodiment of this specification, if the space occupied by the persistently cached data on the local storage device is less than the persistent cache threshold, then caching some or all of the metadata belonging to the metadata in the backup cache area in memory to the local storage device, and transferring the metadata of the cached data from the backup cache area in memory to the cache area, includes: If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, then some or all of the metadata belonging to the metadata in the backup cache area in memory will be cached from the shared data layer to the local storage device, and the metadata of the cached data will be transferred from the backup cache area in memory to the cache area.
[0008] In one possible embodiment of this specification, if the space occupied by the persistently cached data on the local storage device is less than the persistent cache threshold, then caching some or all of the metadata belonging to the metadata in the memory cache area to the local storage device includes: If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is less than the remaining space in the space used for persistent caching on the local storage device, then all the data to which the metadata in the backup cache area belongs will be cached on the local storage device. If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is greater than the remaining space in the space used for persistent caching on the local storage device, then the data to which some of the metadata in the backup cache area belongs will be cached on the local storage device.
[0009] In one possible embodiment of this specification, the backup cache includes a recently accessed list and a frequently accessed list; caching the data to which the metadata in the backup cache belongs to the local storage device includes: Based on the principle of prioritizing the transfer of data belonging to the metadata in the frequently accessed list, some data belonging to the metadata in the backup cache area are cached to the local storage device.
[0010] In one possible embodiment of this specification, the metadata includes the access frequency of the data to which it belongs; caching the data to which the metadata in the backup cache belongs to the local storage device includes: Based on the principle of prioritizing the transfer of data with high access frequency, some metadata belonging to the backup cache area is cached to the local storage device.
[0011] In one possible embodiment of this specification, transferring the metadata of the data deleted from memory from the cache to the backup cache includes: If the remaining space in the backup cache in memory is less than the space required for the metadata of the deleted data, then the space allocated to the backup cache in memory is adjusted to be no less than the space required for the metadata of the deleted data, and the metadata of the deleted data is transferred from the cache to the backup cache.
[0012] In one possible embodiment of this specification, if the space occupied by the persistently cached data on the local storage device is less than the persistent cache threshold, then caching some or all of the metadata belonging to the metadata in the backup cache area in memory to the local storage device, and transferring the metadata of the cached data from the backup cache area in memory to the cache area, includes: If an extension node is added to the database node so that the space occupied by the persistent cached data on the local storage device of the database node is less than the persistent cache threshold, then the metadata in the backup cache area in the memory of the database node is sent to the extension node, so that the extension node caches the data to which some or all of the received metadata belongs on the local storage device, adds the metadata of the cached data to the cache area in memory, and adds the remaining metadata to the backup cache area in memory.
[0013] According to a second aspect of one or more embodiments of this specification, a data caching device is provided for use in a database node, the device comprising: The shrinking module is used to delete some data in the persistent cache on the local storage device if the space occupied by the persistent cache data on the local storage device is greater than the persistent cache threshold, so that the space occupied by the persistent cache data on the local storage device is not greater than the persistent cache threshold, and to transfer the metadata of the deleted data from the cache area in memory to the backup cache area. The expansion module is used to cache some or all of the metadata in the backup cache area in memory to the local storage device if the space occupied by the persistent cache data on the local storage device is less than the persistent cache threshold, and to transfer the metadata of the cached data from the backup cache area in memory to the cache area. The persistent cache threshold is the space threshold allocated by the database node to the local storage device for persistent caching.
[0014] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.
[0015] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in the first aspect by running the executable instructions.
[0016] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.
[0017] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: In the data caching method provided in the embodiments of this specification, the database node is configured with a cache area and a backup cache area in memory. When persisting cached data, the database node stores the data on a local storage device, such as a disk, and loads the metadata of the cached data into the cache area. When the space used for persistent caching in the local storage device shrinks, some data in the already persistently cached data can be deleted, and the metadata of the deleted data can be transferred from the cache area to the backup cache area. When the space used for persistent caching in the local storage device expands, the data to which the metadata in the backup cache area belongs can be persistently cached on the local storage device, and the metadata of the cached data can be transferred from the backup cache area to the cache area. It is evident that the data to which the metadata in the backup cache area belongs is data that the database node wants to cache but cannot cache due to limited space. Recording the metadata of this data allows for targeted addition of persistent cache data when space increases, thereby ensuring a higher hit rate for the added data. This avoids the low hit rate problem caused by randomly adding persistent cache data when space increases in related technologies. The higher hit rate results in lower latency, higher efficiency, and better performance for the database node's data service. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the structure of a data system provided in an exemplary embodiment.
[0019] Figure 2This is a flowchart of a data caching method provided in an exemplary embodiment.
[0020] Figure 3 This is a flowchart of a data caching method provided in another exemplary embodiment.
[0021] Figure 4 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.
[0022] Figure 5 This is a block diagram of a data caching device provided in an exemplary embodiment. Detailed Implementation
[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0024] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0025] First, some of the concepts involved in this instruction manual will be explained.
[0026] Persistent caching refers to storing cached data in persistent storage media so that the data can be reloaded and used after the program is shut down or power is lost. Specifically, in a shared storage architecture, database nodes need to load some frequently accessed data into local memory for caching to avoid the latency caused by the database node having to load data from shared storage in real time every time it accesses data. However, the amount of frequently accessed data that needs to be cached is large, and memory space is very limited and has many uses. Therefore, persistent caching is achieved by persistently storing the frequently accessed data of the shared data layer on persistent storage media such as the local disk of the database node.
[0027] Space elastic scaling: refers to the ability of database nodes to automatically adjust the size of the space used for persistent caching in persistent storage media as needed, thereby improving the flexibility of persistent media space usage.
[0028] Object storage: Object storage is a data management architecture that efficiently and flexibly stores and retrieves massive amounts of unstructured data by treating data as objects rather than files or blocks. Its "everything is an object" concept emphasizes that all data, whether documents, images, videos, or logs, can be processed as objects, which helps improve access efficiency and scalability.
[0029] In related technologies, when a database node in a shared storage architecture shrinks (i.e., when the disk space allocated for persistent caching decreases), redundant data within the already persistently cached data is deleted, and its metadata is deleted synchronously. Conversely, when a database node in a shared storage architecture expands (i.e., when the disk space allocated for persistent caching increases), some data from the shared data layer is randomly added to the local disk. This results in a low hit rate when the database node accesses data; that is, the frequently accessed data is often not pre-cacheed on the local disk. In this case, the database node needs to load the required data from the shared data layer in real time, leading to low data access efficiency and poor database node performance.
[0030] Based on the above-mentioned technical problems, at least one embodiment of this specification provides a data caching method. This method can be applied to database nodes, such as any database node in a database system with a shared storage architecture. When the space used for persistent cache data in the local storage device of the database node shrinks or expands, the method can correspondingly increase or decrease the data in the persistent cache. Furthermore, during the dynamic and elastic change of the space used for persistent cache data, the method can ensure that the data access of the database node has a high hit rate, thereby improving the data service performance of the database node.
[0031] Please refer to the appendix. Figure 1 The shared storage architecture database system may include a shared data layer and at least one database node communicating with it, preferably multiple database nodes. The shared data layer may be in the form of object storage, etc., and stores user data and internal system metadata (such as baselines, dumps, logs, etc.) for sharing among multiple database nodes. Database nodes have local storage devices and memory. The local storage device is a persistent storage medium, such as a disk, used to persistently cache some data within the shared storage, and it is desired that the cached data is data subsequently accessed by the database node, or even frequently accessed data. Memory is a non-persistent storage medium used to store metadata of the persistently cached data. Metadata may include key, required storage space size, offset, region (i.e., whether it is in the cache area or the backup cache area), access frequency value, etc.
[0032] Each database node exists as an instance or replica of the data system. This data system employs a storage-compute separation architecture based on shared storage. Data stored in the shared data layer avoids occupying local storage space on the database nodes while simultaneously enabling data sharing between different database nodes.
[0033] Please refer to the appendix. Figure 2 The flowchart of the method is illustrated by example, including steps S201 and S202.
[0034] In step S201, if the space occupied by the persistent cache data on the local storage device is greater than the persistent cache threshold, then some of the data in the persistent cache on the local storage device is deleted so that the space occupied by the persistent cache data on the local storage device is not greater than the persistent cache threshold, and the metadata of the deleted data is transferred from the memory cache area to the backup cache area.
[0035] The persistent cache threshold is the space threshold allocated by the database node to the local storage device for persistent caching.
[0036] The system internally includes a cache area and a backup cache area, both used to store data metadata. The cache area contains a recently accessed list and a frequently accessed list. Metadata within the cache area is categorized into these two lists. If the data to which metadata belongs has been accessed most recently, its metadata is placed in the recently accessed list; if the data to which metadata belongs is frequently accessed, or if it is both recently accessed and frequently accessed, its metadata is placed in the frequently accessed list. The backup cache area also contains the recently accessed list and the frequently accessed list, and metadata within the backup cache area is categorized into these two lists.
[0037] For example, when deleting some data from the persistent cache on the local storage device in this step, priority can be given to deleting data whose metadata is in the most recently accessed list. As another example, when deleting some data from the persistent cache on the local storage device in this step, priority can be given to deleting data with lower access frequency values.
[0038] The space occupied by persistent cached data on the local storage device, as mentioned in this step, often exceeds the persistent cache threshold due to database node shrinkage, meaning the database node lowers the persistent cache threshold. This is because the database node strictly controls the space occupied by persistent cached data during data persistence caching to ensure it does not exceed the persistent cache threshold. Specifically, this method can control the space occupied by persistent cached data to not exceed the persistent cache threshold using the following cache eviction mechanism: If the space required by the currently cached data exceeds the remaining space on the local storage device used for persistent caching, then some data in the persistent cache on the local storage device is deleted, and the currently cached data is cached on the local storage device. The metadata of the deleted data is then transferred from the cache area in memory to the backup cache area. In this scenario, data with lower access frequency values in the persistent cache can be prioritized for deletion. Alternatively, data whose metadata is in the most recently accessed list can be prioritized for deletion.
[0039] As can be seen from the operation in this step, the data to which the metadata in the backup cache area belongs is not persistently cached on the local storage device. That is, the data to which the metadata in the backup cache area belongs is data that the database node wants to persistently cache to this storage device, but cannot cache due to its limited space. In other words, it is data that is likely to be accessed frequently but is being forced to be discarded.
[0040] For example, in this step, when metadata is transferred from the cache area to the backup cache area, metadata in the latest access list in the cache area can be transferred to the latest access list in the backup cache area, and metadata in the frequently accessed list in the cache area can be transferred to the frequently accessed list in the backup cache area.
[0041] For example, in this step, when transferring metadata from the cache area to the standby cache area, if the remaining space in the standby cache area in memory is less than the space required for the metadata of the deleted data, then the space allocated to the standby cache area in memory is adjusted to be no less than the space required for the metadata of the deleted data, and the metadata of the deleted data is transferred from the cache area to the standby cache area. Since the total amount of metadata remains unchanged during the above transfer process, and the storage space in memory used to load the metadata also remains unchanged, the standby cache area can be expanded by adjusting the ratio between the cache area and the standby cache area, so that all the metadata of the deleted data can be retained in memory. As mentioned earlier, the metadata in the standby cache area belongs to data that the database node wants to persistently cache to the current storage device but cannot cache due to limited space. This example allows the database node to record as much data as possible that it wants to cache.
[0042] In step S402, if the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, then some or all of the metadata in the backup cache area in memory is cached to the local storage device, and the metadata of the cached data is transferred from the backup cache area in memory to the cache area.
[0043] As mentioned earlier, the metadata in the backup cache area belongs to the data that the database node wants to persist and cache to the current storage device but cannot cache due to limited space. In step S401, when the database node is scaled down (i.e., the persistent cache threshold is reduced), the metadata of this part of the data is loaded, so that the data that needs to be added to the persistent cache in this step is targeted, and the access hit rate of the database node is increased. In particular, compared with the method of randomly selecting data to persist and cache, it overcomes the blindness and increases the access hit rate of the database node.
[0044] In a shared storage system architecture, this step may include: if the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, then the data to which some or all of the metadata in the backup cache area in memory belongs is cached from the shared data layer to the local storage device, and the metadata of the cached data (i.e. the newly added cached data in this step) is transferred from the backup cache area in memory to the cache area.
[0045] It should be understood that if metadata is stored in the in-memory backup cache, it means that the database node previously recorded data in the backup cache due to scaling down. If metadata is not stored in the in-memory backup cache, it means that the database node previously did not record data in the backup cache due to scaling down.
[0046] This step addresses a scenario where a database node is scaled up again after a period of time following a scaling down. In this case, the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold because the scaling up causes a sudden increase in the persistent cache threshold, and metadata is stored in the backup cache area. Therefore, when caching some or all of the metadata from the backup cache area in memory to the local storage device, this step can be performed as shown in the following two optional examples.
[0047] Optional Example 1 If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is not greater than the remaining space in the space used for persistent caching on the local storage device, then all the data to which the metadata in the backup cache area belongs will be cached on the local storage device.
[0048] Optional Example 2 If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is greater than the remaining space in the space used for persistent caching on the local storage device, then the data to which some of the metadata in the backup cache area belongs will be cached on the local storage device.
[0049] For example, in this example, data belonging to some metadata in the backup cache area can be cached to the local storage device based on the principle of prioritizing the transfer of data to the metadata in the frequently accessed list.
[0050] For example, in this case, the data to which some metadata in the backup cache belongs can be cached to the local storage device, based on the principle of prioritizing the transfer of data with high access frequency.
[0051] If the metadata is not stored in the memory cache area, it means that the remaining persistent cache space in the local storage device of the database node is not completely filled after the database node starts running. In this case, you only need to perform the general data persistent caching method, that is, cache some data in the shared data layer to the local storage device of the database node to gradually fill its persistent cache space, and add the metadata of the persistent cache data to the memory cache area.
[0052] When scaling up database nodes, two methods can be used: vertical scaling and horizontal scaling. Vertical scaling refers to adding local storage devices on a single machine for persistent caching. Horizontal scaling refers to adding expansion nodes to the database node to increase local storage devices, and using some or all of the space of the newly added local storage devices to increase the space for persistent caching.
[0053] For example, vertical scaling can be performed by persistently caching data in the manner described in the above embodiments.
[0054] For example, horizontal scaling can further complete the persistent caching of data in the following manner: If an extension node is added to the database node so that the space occupied by the persistently cached data on the local storage device of the database node is less than the persistent caching threshold, then the metadata in the backup cache area in the memory of the database node is sent to the extension node, so that the extension node caches the data to which some or all of the received metadata belongs on the local storage device, adds the metadata of the cached data to the cache area in memory, and adds the remaining metadata to the backup cache area in memory.
[0055] Compared to vertical scaling, horizontal scaling requires the original node to share the metadata in the backup cache area in memory to the newly added extension node, so that the extension node can selectively persist and cache some data in the shared data layer based on the metadata in the backup cache area.
[0056] For further details regarding horizontal scaling, they can be completed in accordance with the methods described in the above embodiments: Caching a portion of the data from the shared data layer to local storage devices.
[0057] If the space required for all the metadata received by the extended node is no greater than the space used for persistent caching on the extended node's local storage device, then the data to which all the received metadata belongs will be cached on the local storage device.
[0058] If the space required for all the metadata received by the extended node exceeds the space available for persistent caching on the extended node's local storage device, then a portion of the received metadata will be cached on the local storage device. Preferably, data belonging to metadata in the backup cache area will be cached on the local storage device, prioritizing the transfer of data belonging to metadata in the frequently accessed list. Preferably, data belonging to metadata in the backup cache area will be cached on the local storage device, prioritizing the transfer of data with high access frequency.
[0059] Please refer to the appendix. Figure 3 The following example illustrates the data caching method derived from the above embodiments, including the following steps: Persistent cache space allocation: The persistent cache threshold on the local disk of the database node is 100GB, and the cache area in memory is 100GB, which is the space required for the metadata of the persistent cache data.
[0060] Data persistence cache: As the data access database nodes gradually fill up the space on the local disk used for persistence cache, that is, when 100GB of persistent cache data is written.
[0061] Persistent cache space reduction: The database node is reduced in size, and the persistent cache threshold is lowered to 20GB. Then, 80GB of data out of the 100GB persistent cache data is evicted, i.e. deleted. The memory cache area is configured to be the space required for the metadata of the 20GB persistent cache data, the standby cache area is configured to be the space required for the metadata of the 80GB persistent cache data, and the metadata of the deleted 80GB data is transferred from the cache area to the standby cache area.
[0062] Persistent cache space expansion: When the database node is expanded, increasing the persistent cache threshold to 100GB, 80GB of data containing metadata in the backup cache area will be persistently cached to the local disk to reach the persistent cache threshold. For example, data to be persistently cached to the local disk can be selected using a key from the metadata in the backup cache area.
[0063] In this process, the metadata of deleted data in the shrinking scenario is recorded in the memory's backup cache area. In the expanding scenario, the metadata recorded in the backup cache area can be used to selectively persist cached data, overcoming the blindness and randomness of related technologies and improving the hit rate of database node access. Moreover, when recording metadata in the backup cache area, the space of the backup cache area is expanded to record as much metadata as possible, so that there is enough metadata to guide the selection of data for persistent caching in the expanding scenario.
[0064] In the data caching method provided in the embodiments of this specification, the database node is configured with a cache area and a backup cache area in memory. When persisting cached data, the database node stores the data on a local storage device, such as a disk, and loads the metadata of the cached data into the cache area. When the space used for persistent caching in the local storage device shrinks, some data in the already persistently cached data can be deleted, and the metadata of the deleted data can be transferred from the cache area to the backup cache area. When the space used for persistent caching in the local storage device expands, the data to which the metadata in the backup cache area belongs can be persistently cached on the local storage device, and the metadata of the cached data can be transferred from the backup cache area to the cache area. It is evident that the data to which the metadata in the backup cache area belongs is data that the database node wants to cache but cannot cache due to limited space. Recording the metadata of this data allows for targeted addition of persistent cache data when space increases, thereby ensuring a higher hit rate for the added data. This avoids the low hit rate problem caused by randomly adding persistent cache data when space increases in related technologies. The higher hit rate results in lower latency, higher efficiency, and better performance for the database node's data service.
[0065] Figure 4 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 4At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0066] Please refer to Figure 5 Data caching devices can be applied to, for example Figure 4 The database node running on the device shown is used to implement the technical solution described in this specification. The data caching device may include: The shrinkage module 501 is used to delete part of the data in the persistent cache on the local storage device if the space occupied by the data in the persistent cache on the local storage device is greater than the persistent cache threshold, so that the space occupied by the data in the persistent cache on the local storage device is not greater than the persistent cache threshold, and to transfer the metadata of the deleted data from the cache area in memory to the backup cache area. The expansion module 502 is used to cache some or all of the metadata in the backup cache area in memory to the local storage device if the space occupied by the persistent cache data on the local storage device is less than the persistent cache threshold, and to transfer the metadata of the cached data from the backup cache area in memory to the cache area. The persistent cache threshold is the space threshold allocated by the database node to the local storage device for persistent caching.
[0067] In one embodiment of this specification, the device further includes an elimination module for: If the space required by the data to be cached is greater than the remaining space in the space used for persistent caching on the local storage device, then some data in the persistent cache data on the local storage device is deleted, so that the data to be cached is cached in the local storage device, and the metadata of the deleted data is transferred from the cache area in memory to the backup cache area.
[0068] In one embodiment of this specification, the expansion module is used for: If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, then some or all of the metadata belonging to the metadata in the backup cache area in memory will be cached from the shared data layer to the local storage device, and the metadata of the cached data will be transferred from the backup cache area in memory to the cache area.
[0069] In one embodiment of this specification, the expansion module is used to cache some or all of the metadata belonging to the metadata in the memory cache area to the local storage device if the space occupied by the persistent cache data on the local storage device is less than the persistent cache threshold. The module is configured to: If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is less than the remaining space in the space used for persistent caching on the local storage device, then all the data to which the metadata in the backup cache area belongs will be cached on the local storage device. If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is greater than the remaining space in the space used for persistent caching on the local storage device, then the data to which some of the metadata in the backup cache area belongs will be cached on the local storage device.
[0070] In one embodiment of this specification, the backup cache includes a recently accessed list and a frequently accessed list; when the expansion module caches data to which some of the metadata in the backup cache belongs to the local storage device, it is used for: Based on the principle of prioritizing the transfer of data belonging to the metadata in the frequently accessed list, some data belonging to the metadata in the backup cache area are cached to the local storage device.
[0071] In one embodiment of this specification, the metadata includes the access frequency of the data to which it belongs; when the expansion module is used to cache the data to which the metadata in the backup cache area belongs to the local storage device, it is used for: Based on the principle of prioritizing the transfer of data with high access frequency, some metadata belonging to the backup cache area is cached to the local storage device.
[0072] In one embodiment of this specification, when the expansion module is used to transfer the metadata of deleted data from the cache area to the backup cache area, it is used for: If the remaining space in the backup cache in memory is less than the space required for the metadata of the deleted data, then the space allocated to the backup cache in memory is adjusted to be no less than the space required for the metadata of the deleted data, and the metadata of the deleted data is transferred from the cache to the backup cache.
[0073] In one embodiment of this specification, the expansion module is used for: If an extension node is added to the database node so that the space occupied by the persistent cached data on the local storage device of the database node is less than the persistent cache threshold, then the metadata in the backup cache area in the memory of the database node is sent to the extension node, so that the extension node caches the data to which some or all of the received metadata belongs on the local storage device, adds the metadata of the cached data to the cache area in memory, and adds the remaining metadata to the backup cache area in memory.
[0074] More details about the various modules in this device have been described in detail in the first aspect of the data caching method, and will not be repeated here.
[0075] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0076] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0078] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0080] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0081] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0082] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.
[0083] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."
[0084] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.
Claims
1. A data caching method applied to a database node, the database node including a local storage device and memory; the local storage device being used for persistently caching data in a shared data layer; the memory including a cache area for storing metadata of the persistently cached data on the local storage device, and a backup cache area for storing metadata of data deleted from the persistently cached data on the local storage device; the method comprising: If the space occupied by the persistent cache data on the local storage device is greater than the persistent cache threshold, then some of the data in the persistent cache on the local storage device will be deleted so that the space occupied by the persistent cache data on the local storage device is not greater than the persistent cache threshold, and the metadata of the deleted data will be transferred from the memory cache area to the backup cache area. If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold and there is metadata in the backup cache area in memory, then the data to which some or all of the metadata in the backup cache area in memory belongs will be persistently cached from the shared data layer to the local storage device, and the metadata of the cached data will be transferred from the backup cache area in memory to the cache area. The persistent cache threshold is the space threshold allocated by the database node on the local storage device for persistently caching data in the shared data layer.
2. The data caching method according to claim 1, further comprising: If the space required by the data to be cached is greater than the remaining space in the space used for persistent caching on the local storage device, then some data in the persistent cache data on the local storage device is deleted, so that the data to be cached is cached in the local storage device, and the metadata of the deleted data is transferred from the cache area in memory to the backup cache area.
3. The data caching method according to claim 1, wherein if the space occupied by the persistently cached data on the local storage device is less than the persistent cache threshold, then the data to which some or all of the metadata in the memory cache area belongs is persistently cached from the shared data layer to the local storage device, includes: If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is less than the remaining space in the space used for persistent caching on the local storage device, then all the data to which the metadata in the backup cache area belongs will be persistently cached from the shared data layer to the local storage device. If the space occupied by the persistent cached data on the local storage device is less than the persistent cache threshold, the backup cache area in memory contains metadata, and the space required by the data to which the metadata in the backup cache area belongs is greater than the remaining space in the space used for persistent caching on the local storage device, then the data to which some of the metadata in the backup cache area belongs will be persistently cached from the shared data layer to the local storage device.
4. The data caching method according to claim 3, wherein the backup cache area includes a recently accessed list and a frequently accessed list; the step of caching the data to which the metadata in the backup cache area belongs to the local storage device includes: Based on the principle of prioritizing the transfer of data belonging to the metadata in the frequently accessed list, some data belonging to the metadata in the backup cache area are cached to the local storage device.
5. The data caching method according to claim 3, wherein the metadata includes the access frequency of the data to which it belongs; the step of caching the data to which the metadata in the backup cache area belongs to the local storage device includes: Based on the principle of prioritizing the transfer of data with high access frequency, some metadata belonging to the backup cache area is cached to the local storage device.
6. The data caching method according to claim 1, wherein transferring the metadata of the deleted data in memory from the cache area to the backup cache area includes: If the remaining space in the backup cache in memory is less than the space required for the metadata of the deleted data, then the space allocated to the backup cache in memory is adjusted to be no less than the space required for the metadata of the deleted data, and the metadata of the deleted data is transferred from the cache to the backup cache.
7. The data caching method according to claim 1, wherein if the space occupied by the persistently cached data on the local storage device is less than the persistent cache threshold, then the data to which some or all of the metadata in the backup cache area in memory belongs is persistently cached from the shared data layer to the local storage device, and the metadata of the cached data is transferred from the backup cache area in memory to the cache area, comprising: If an extension node is added to the database node so that the space occupied by the persistently cached data on the local storage device of the database node is less than the persistent cache threshold, then the metadata in the backup cache area in the memory of the database node is sent to the extension node, so that the extension node will persistently cache the data to which some or all of the received metadata belongs from the shared data layer to the local storage device, add the metadata of the cached data to the cache area in memory, and add the remaining metadata to the backup cache area in memory.
8. A data caching device applied to a database node, the database node including a local storage device and memory; the local storage device being used to persistently cache data in a shared data layer; the memory including a cache area for storing metadata of the persistently cached data on the local storage device, and a backup cache area for storing metadata of data deleted from the persistently cached data on the local storage device; The device includes: The shrinking module is used to delete some data in the persistent cache on the local storage device if the space occupied by the persistent cache data on the local storage device is greater than the persistent cache threshold, so that the space occupied by the persistent cache data on the local storage device is not greater than the persistent cache threshold, and to transfer the metadata of the deleted data from the cache area in memory to the backup cache area. The expansion module is used to persistently cache some or all of the data belonging to the metadata in the backup cache area in memory to the local storage device if the space occupied by the persistent cache data on the local storage device is less than the persistent cache threshold and the backup cache area in memory contains metadata. The module also transfers the metadata of the cached data from the backup cache area in memory to the cache area. The persistent cache threshold is the space threshold allocated by the database node on the local storage device for persistently caching data in the shared data layer.
9. The data caching device according to claim 8, further comprising an eviction module, configured to: If the space required by the data to be cached is greater than the remaining space in the space used for persistent caching on the local storage device, then some data in the persistent cache data on the local storage device is deleted, so that the data to be cached is cached in the local storage device, and the metadata of the deleted data is transferred from the cache area in memory to the backup cache area.
10. The data caching device according to claim 8, wherein the backup cache area includes a recently accessed list and a frequently accessed list; the expansion module, when caching data to which some metadata in the backup cache area belongs to the local storage device, is used for: Based on the principle of prioritizing the transfer of data belonging to the metadata in the frequently accessed list, some data belonging to the metadata in the backup cache area are cached to the local storage device.
11. The data caching device according to claim 8, wherein the metadata includes the access frequency of the data to which it belongs; the expansion module, when caching the data to which the metadata in the backup cache area belongs to the local storage device, is used for: Based on the principle of prioritizing the transfer of data with high access frequency, some metadata belonging to the backup cache area is cached to the local storage device.
12. The data caching device according to claim 8, wherein when the expansion module is used to transfer the metadata of deleted data from the cache area to the backup cache area, it is used for: If the remaining space in the backup cache in memory is less than the space required for the metadata of the deleted data, then the space allocated to the backup cache in memory is adjusted to be no less than the space required for the metadata of the deleted data, and the metadata of the deleted data is transferred from the cache to the backup cache.
13. The data caching device according to claim 8, wherein the expansion module is used for: If an extension node is added to the database node so that the space occupied by the persistently cached data on the local storage device of the database node is less than the persistent cache threshold, then the metadata in the backup cache area in the memory of the database node is sent to the extension node, so that the extension node will persistently cache the data to which some or all of the received metadata belongs from the shared data layer to the local storage device, add the metadata of the cached data to the cache area in memory, and add the remaining metadata to the backup cache area in memory.
14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.
15. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 7 by executing the executable instructions.
16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
Metadata storage method, device and apparatus, and computer-readable storage medium
CN109376125A
DFS-oriented client persistent caching method and system in global namespace
CN110750507A
