Catalog level data cache management method, client, server, system and medium

Abstract

The application relates to the technical field of distributed file systems, and provides a directory-level data cache management method, a client, a server, a system and a medium, which comprises the following steps: obtaining an effective cache strategy of a target file under a target directory from a metadata server; when an access request for the target file is received, mapping a file offset range indicated by the access request into an object range of at least one data object according to file layout information of the target file; in the case that data corresponding to the object range is loaded into a cache entry of a local memory cache, setting a cache protection mark for the cache entry according to the effective cache strategy; and in the process of performing cache recovery on the local memory cache, skipping or postponing the recovery of the cache entry provided with the cache protection mark. By introducing the directory-level effective cache strategy and setting the cache protection mark locally on the client, the application realizes the cache protection of specific key file data, so that the access delay jitter is reduced.

Description

Technical Field

[0001] This invention relates to the field of distributed file system technology, and particularly to data caching management technology in distributed file systems, especially to a directory-level data caching management method, client, server, system, and medium. Background Technology

[0002] With the development of distributed storage technology, distributed file systems are widely used in scenarios involving massive data storage and high-concurrency access. To improve file data access performance, existing distributed file systems, such as the Ceph File System (CephFS), typically introduce data caching mechanisms on the client or server side, such as using object caching components or the operating system's page cache to temporarily store accessed data.

[0003] However, existing data caching mechanisms, primarily based on global capacity limits and global eviction algorithms, have limitations when facing complex business scenarios. Specifically, when a large volume of ordinary data is read, even critical business data (such as AI model files, index files, and hotspot configurations) that are crucial to the system may be evicted from the cache due to the global eviction policy. When the business accesses this critical data again, it must be retrieved from remote storage nodes over the network, resulting in severe access latency jitter and failing to meet the needs of high-frequency businesses sensitive to cache jitter. Furthermore, existing cache configurations often only control the global cache size and cannot provide cache retention strategies for specific ranges or types of critical datasets. Summary of the Invention

[0004] This invention provides a directory-level data cache management method, client, server, system, and medium to address the shortcomings of the global cache eviction mechanism in distributed file systems in related technologies, which cannot provide fine-grained protection for specific critical data, leading to critical data being easily squeezed out of the cache by irrelevant data and causing access latency jitter.

[0005] This invention provides a directory-level data caching management method, applied to a distributed file system client, comprising: Obtain the effective caching strategy for the target file in the target directory from the metadata server. The effective caching strategy is determined and issued by the metadata server based on the caching persistence strategy configured for the target directory. Upon receiving an access request for the target file, the file offset range indicated by the access request is mapped to the object range of at least one data object based on the file layout information of the target file. When the data corresponding to the object range is loaded into the cache entry of the local memory cache, or when the cache entry of the data corresponding to the object range is hit in the local memory cache, a cache protection flag is set for the cache entry according to the effective caching strategy; During the cache reclamation process of the local memory cache, cache entries with the cache protection flag set are skipped or the reclamation is delayed.

[0006] According to a directory-level data caching management method provided by the present invention, the step of obtaining an effective caching strategy for target files in a target directory from a metadata server includes: If any one of the following operations is performed on the target file: path resolution, opening, or first read, or if a cache permission grant message or client session message for the target file is received, the effective caching policy issued by the metadata server shall be received. The effective caching strategy is recorded in the inode or file handle corresponding to the target file. The effective caching strategy includes at least one of the following: strategy identifier, strategy version, cache priority, and capacity limit.

[0007] According to a directory-level data cache management method provided by the present invention, the step of setting a cache protection flag for the cache entry according to the effective caching strategy includes: Determine the file to which the cache entry belongs and the effective caching strategy corresponding to the file based on the object scope; The description structure of the cache entry records the protection identifier, the policy identifier of the effective caching policy, and the directory identifier of the target directory, wherein the cache entry includes buffered data entries, object entries, or page cache entries in the object cache.

[0008] According to a directory-level data cache management method provided by the present invention, the step of skipping or delaying the reclamation of cache entries marked with the cache protection flag during the cache reclamation process of the local memory cache includes: If the local memory cache usage exceeds a preset threshold, candidate cache entries to be reclaimed are determined from the local memory cache according to a preset cache eviction policy. If the candidate cache entry has the cache protection flag set, skip the recycling of the candidate cache entry, or migrate the candidate cache entry to the protected cache queue; If the candidate cache entry does not have the cache protection flag set, the candidate cache entry is reclaimed.

[0009] According to a directory-level data cache management method provided by the present invention, the effective caching strategy includes a caching mode, and the caching mode includes a persistent mode and a priority mode; The step of skipping or delaying the reclamation of cache entries marked with the cache protection flag during the cache reclamation process of the local memory cache includes: If the local memory cache is insufficient, cache reclamation is performed according to the directory-aware hierarchical order. The directory-aware hierarchical order is as follows: first, cache entries that have not been marked with the cache protection flag are reclaimed; if the target release capacity has not been reached, cache entries with the cache mode set to the priority mode are reclaimed; if the target release capacity has not been reached and the degradation conditions are met, cache entries with the cache mode set to the resident mode are reclaimed.

[0010] According to a directory-level data caching management method provided by the present invention, the effective caching strategy further includes the maximum cache capacity of the target directory. After the step of setting a cache protection flag for the cache entry, the method further includes: If the maximum cache capacity is non-zero and the protected cache capacity of the target directory exceeds the maximum cache capacity, the protected cache entry corresponding to the target directory is reclaimed according to at least one of access time, file priority, or object offset within the protected cache entry corresponding to the target directory.

[0011] A directory-level data cache management method provided by the present invention further includes: If the degradation condition is met, the cache protection flag of the cache entry is removed, or the cache mode of the cache entry is adjusted from the resident mode to the priority mode; The downgrade condition is any one of the following: the maximum cache capacity is a non-zero value and the protected cache capacity corresponding to the target directory exceeds the maximum cache capacity; the available memory of the client is lower than the security threshold; a forced release instruction is received; the cache resident policy of the target directory is deleted or disabled; the data of the target file is modified, truncated, deleted, or its consistency fails; or the policy version of the local record is determined to be expired after reconnecting with the metadata server.

[0012] According to a directory-level data cache management method provided by the present invention, after the step of setting a cache protection flag for the cache entry, the method further includes: If a truncation event, deletion event, renaming event, cache permission revocation event, or snapshot rollback event is detected for the target file, the corresponding cache entries are cleaned up or the cache protection flag of the corresponding cache entries is removed, based on the inode corresponding to the target file and the object scope.

[0013] According to a directory-level data cache management method provided by the present invention, after the step of mapping the file offset range indicated by the access request to the object range of at least one data object, the method further includes: When the effective caching policy indicates that server-side cache protection is enabled, send an object read / write request carrying an object-level cache protection tag to the object storage service provider, or send an independent object protection request to the object storage service provider. The object-level cache protection tag or the object protection request is determined based on the object scope and the effective caching policy. It is used to instruct the object storage service to set a server-side cache protection tag for the server-side cache entries corresponding to the object scope, and to skip or delay the recycling of server-side cache entries with the server-side cache protection tag during the server-side cache recycling process.

[0014] This invention also provides a directory-level data cache management method, applied to a metadata server of a distributed file system, comprising: Receive a policy setting request for a target directory, and obtain the cache persistence policy configured for the target directory from the policy setting request; Based on the cache resident policy, determine the effective caching policy for the target files in the target directory; An effective caching policy for the target file is issued to the client, so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to the object range of at least one data object according to the file layout information of the target file. If the data corresponding to the object range is loaded into the cache entry of the client's local memory cache, or if the cache entry of the data corresponding to the object range is hit in the local memory cache, the client sets a cache protection flag for the cache entry according to the effective caching policy. The client is used to skip or postpone the reclamation of cache entries with the cache protection flag set during the cache reclamation process of the local memory cache.

[0015] According to a directory-level data caching management method provided by the present invention, after the step of obtaining the cache resident policy configured for the target directory from the policy setting request, the method further includes: The validity of the policy fields in the cache persistence policy is verified. The policy fields include at least one of the following: server-side cache protection enabled status, client-side cache protection enabled status, reserved cache capacity, maximum cache capacity, cache priority, subdirectory inheritance identifier, cache mode, and cache eviction policy. The cache persistence policy that has passed the verification is persisted and a policy version corresponding to the cache persistence policy is generated.

[0016] According to the directory-level data cache management method provided by the present invention, after the step of persistently storing the verified cache resident policy, the method further includes: Maintain policy status statistics information corresponding to the cache resident policy. The policy status statistics information includes at least one of the following: client information holding the cache resident policy, protected cache capacity reported by each client, and protected cache capacity reported by the object storage service client.

[0017] According to a directory-level data caching management method provided by the present invention, the step of determining an effective caching strategy for target files in the target directory based on the caching persistence strategy includes: If the target file does not have a caching policy configured, search upwards through the parent directory of the target file for the directory with the most recently configured caching resident policy, and determine the caching resident policy configured in the found directory as the effective caching policy for the target file.

[0018] A directory-level data cache management method provided by the present invention further includes: If the cache persistence policy of the target directory is added, modified or deleted, update the policy version corresponding to the cache persistence policy and determine the list of clients that currently hold the inode or cache permissions of the files in the target directory. A policy update notification is sent to each client in the client list so that each client updates its effective caching policy for local records.

[0019] The present invention also provides a directory-level data cache management device, applied to a client of a distributed file system, comprising: The policy acquisition unit is used to obtain the effective caching policy of the target file in the target directory from the metadata server. The effective caching policy is determined and issued by the metadata server according to the caching persistence policy configured for the target directory. An object mapping unit is used to, upon receiving an access request for the target file, map the file offset range indicated by the access request to the object range of at least one data object based on the file layout information of the target file. A cache tagging unit is used to set a cache protection tag for a cache entry according to the effective caching strategy when the data corresponding to the object range is loaded into a cache entry in the local memory cache, or when a cache entry corresponding to the object range is hit in the local memory cache. A cache reclamation unit is used to skip or postpone the reclamation of cache entries with the cache protection flag during the process of performing cache reclamation on the local memory cache.

[0020] The present invention also provides a directory-level data cache management device, applied to a metadata server of a distributed file system, comprising: The request receiving unit is used to receive a policy setting request for a target directory and obtain the cache persistence policy configured for the target directory from the policy setting request. The strategy determination unit is used to determine the effective caching strategy for the target files in the target directory based on the cache resident strategy. The policy distribution unit is used to distribute an effective caching policy for the target file to the client, so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to the object range of at least one data object according to the file layout information of the target file. If the data corresponding to the object range is loaded into a cache entry in the client's local memory cache, or if a cache entry for the data corresponding to the object range is hit in the local memory cache, the client sets a cache protection flag for the cache entry according to the effective caching policy. The client is used to skip or postpone the reclamation of cache entries with the cache protection flag during the cache reclamation process of the local memory cache.

[0021] The present invention also provides a client, including a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor executes the computer program to implement the directory-level data cache management method applied to the client as described above.

[0022] The present invention also provides a metadata server, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the directory-level data cache management method applied to the metadata server as described above.

[0023] The present invention also provides a distributed file system, including a metadata server as described above, and at least one client as described above.

[0024] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the directory-level data cache management method as described above.

[0025] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the directory-level data cache management method as described above.

[0026] The present invention provides a directory-level data caching management method, client, server, system, and medium. The client obtains an effective caching strategy for target files in the target directory from a metadata server, transforming traditional global indiscriminate caching control into fine-grained directory-level caching control. When processing actual file access requests, the client uses file layout information to map the file offset range to the object range of the underlying data object. When data is loaded into the local memory cache or when the corresponding data is hit in the local memory cache, a cache protection flag is set for the corresponding cache entry according to the effective caching strategy. This mechanism enables the client to identify protected critical file data when performing cache reclamation operations on the local memory cache, and thus skip or postpone the reclamation of these cache entries with cache protection flags. This effectively avoids data in critical directories being squeezed out of the local memory cache due to a large number of irrelevant data reading operations in the system, thereby reducing the need for repeated cross-network remote reading of critical data and reducing access latency jitter. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is one of the flowcharts illustrating the directory-level data cache management method provided by the present invention; Figure 2 This is the second flowchart illustrating the directory-level data cache management method provided by the present invention; Figure 3 This is a flowchart of the catalog-aware hierarchical elimination strategy provided by the present invention. Figure 4 This is a schematic diagram of the overall process of the directory-level data cache management method for distributed file systems provided by the present invention; Figure 5 This is one of the structural schematic diagrams of the directory-level data cache management device provided by the present invention; Figure 6This is the second schematic diagram of the directory-level data cache management device provided by the present invention; Figure 7 This is a schematic diagram of the structure of the computer device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] With the development of distributed storage technology, distributed file systems (such as CephFS) are widely used in various scenarios requiring massive data storage and high-concurrency access. In the CephFS system, file data is typically stored in the underlying distributed object storage layer, such as RADOS (Reliable Autonomic Distributed Object Store). When a client accesses a file, it maps the file offset to one or more object ranges based on the file layout information in the file inode, and then reads or writes the corresponding object data to the OSD (Object Storage Daemon). Here, the OSD is the service process in the distributed file system responsible for object storage, replication, recovery, and read / write processing.

[0031] To improve access performance, related technologies typically introduce data caching mechanisms on either the client or server side. Taking the CephFS user-space client as an example, it usually caches object data on the client side through an object caching component (ObjectCacher); while kernel clients can also use the operating system's page cache to store accessed data. Specifically, the relevant caching management mechanisms mainly include client metadata caching, client object data caching, operating system page caching, and MDS (Metadata Server) directory pinning mechanisms.

[0032] The client-side metadata cache refers to the client-side maintenance of metadata such as directory entries and inodes in CephFS. This cache is primarily for path resolution and metadata access and does not directly guarantee that file data content resides in the memory cache. The client-side object data cache refers to the object caching mechanism existing in relevant user-space clients (such as libcephfs or ceph-fuse). This cache is constrained by the client's global capacity and typically uses LRU (Least Recently Used) or similar strategies to evict clean cache blocks. The operating system page cache refers to the storage of file data in kernel clients or FUSE (Filesystem in Userspace) scenarios, where file data can be stored in the operating system's page cache. However, this cache is managed uniformly by the operating system, and CephFS typically cannot declare a cache persistence policy for a specific directory. The MDS directory pinning mechanism refers to MDS supporting a specific directory pinning mechanism, but this mechanism is mainly used for load balancing metadata management in MDS and is not for ensuring file data resides in the client-side memory cache. In addition, the RADOS layer has object pinning capabilities related to cache layering, but its goal is only to pin objects in the RADOS cache layer, rather than in the CephFS client's local memory cache, and this mechanism is an old mechanism that is not recommended for new deployments.

[0033] However, the aforementioned cache management mechanism has certain limitations when dealing with business scenarios that are sensitive to cache jitter, such as reading AI (Artificial Intelligence) model files, accessing high-frequency configuration directories, and database and inference services. Specifically, the relevant CephFS cache is mainly managed uniformly based on the client's global capacity and a global LRU eviction mechanism. Administrators cannot directly declare that data in a certain directory must be preferentially retained in the memory cache. When a client reads a large amount of data from other unrelated directories, even if the data in a critical directory is very important to the business (such as model files or index files that require stable, low-latency reading), it is very easy for the ordinary LRU eviction mechanism to clear it from the memory cache. This causes the next time the data in that critical directory is accessed, it needs to go through the remote read path again, resulting in access latency jitter.

[0034] Moreover, client-side cache parameters can only control the overall cache size and cannot set retention limits, maximum limits, or eviction priorities for different directories in a fine-grained manner; while the MDS directory pinning mechanism only affects the ownership of directory metadata and cannot guarantee that the file content under the directory will remain in the client's memory cache.

[0035] Furthermore, when different clients access the same CephFS directory, each client typically manages its data cache independently according to its local caching rules. MDS, on the other hand, mainly manages metadata such as directories and inodes, and cannot uniformly express and distribute data caching strategies such as which directory data should be kept in the memory of a specified client, how much capacity to keep, and what priority to use. As a result, it is impossible to coordinate the data caching behavior of multiple clients on the same directory.

[0036] To address this, the present invention provides a directory-level data cache management method. By introducing the file system semantics of the directory into the data cache eviction decision, a configurable cache resident strategy is provided for a specified directory. This allows cache management to move beyond simply relying on global capacity constraints or global LRU strategies for eviction. Instead, it combines directory-level caching strategies to protect data caches, thereby preventing data caches in critical directories from being evicted by other ordinary data caches, thus reducing access latency jitter and overcoming the aforementioned defects.

[0037] Figure 1 This is one of the flowcharts illustrating the directory-level data cache management method provided by the present invention, such as... Figure 1 As shown, this method is applied to a client of a distributed file system, and the method includes: Step S110: Obtain the effective caching policy for the target file in the target directory from the metadata server. The effective caching policy is determined and issued by the metadata server based on the caching persistence policy configured for the target directory.

[0038] It should be noted that the method provided in this embodiment of the invention is mainly applied to distributed file systems (such as CephFS). Here, the distributed file system architecture typically includes a client, a metadata server (MDS), and an object storage service (such as OSD). The metadata server is mainly responsible for managing metadata information such as directories, file inodes, directory entries, and permissions; the object storage service is mainly responsible for the actual storage, reading, writing, and copying of underlying data objects; and the client is the entry point for business access to the file system.

[0039] The execution subject of the method provided in this embodiment of the invention can be a client of a distributed file system. The client can be a user-space client (e.g., a client implemented based on libcephfs, ceph-fuse, etc.) or a kernel-space client (e.g., a CephFS client in the operating system kernel). This embodiment of the invention does not specifically limit this.

[0040] Specifically, in a distributed file system, administrators or business systems can pre-configure data caching protection rules for specific directories (i.e., target directories). Here, the target directory typically refers to a critical directory in the business scenario that is sensitive to cache jitter, requires stable low-latency reads, or is frequently accessed, such as AI model file directories, index file directories, parameter file directories, frequently accessed configuration directories, or hot data directories. The data caching protection rules configured for this target directory are called cache resident policies. These policies can be set through extended attributes, management commands, configuration files, or APIs (Application Programming Interfaces), declaring to the system that data in this directory should be preferentially retained in the memory cache or reside there as long as possible, thereby preventing it from being evicted by other ordinary data.

[0041] The metadata server manages these directory-level caching persistence policies. When a client's business application needs to access a specific file (i.e., the target file) within that target directory, the metadata server determines the caching policy that the target file should follow—the effective caching policy. The client can obtain this effective caching policy by interacting with the metadata server, such as when resolving file paths or opening files.

[0042] Because the file system has a directory hierarchy, the effective caching strategy for a target file can be determined directly by the metadata server based on the cache resident strategy configured for the target directory, or it can be determined from the cache resident strategies configured in its parent directories based on specific inheritance logic. This embodiment of the invention does not specifically limit this. For example, when the target file itself does not have a directly set caching strategy, the metadata server can search upwards along its parent directories for the directory most recently configured with a cache resident strategy, and use the found strategy as the actual effective caching strategy corresponding to the target file.

[0043] Step S120: Upon receiving an access request for the target file, the file offset range indicated by the access request is mapped to the object range of at least one data object based on the file layout information of the target file.

[0044] Specifically, when a client's business application initiates a read or write operation on the aforementioned target file, an access request for that target file is generated. Typically, the business application's access request is file-based, meaning the request carries a file identifier and the file offset range to be read or written, such as the starting offset and read length within the file.

[0045] However, at the underlying distributed object storage layer (such as the RADOS layer), file data is split into multiple fragments and stored in different data objects. Therefore, the client obtains the file layout information of the target file and, based on this file layout information, converts the file offset range based on the file dimension into the object range of at least one data object that the underlying object storage can recognize.

[0046] In a distributed file system, file layout information (e.g., layout) refers to the specific rule parameters that determine how file data is physically divided and mapped to the underlying distributed object storage layer. This typically includes information such as block size, number of blocks, and object size. Data objects (e.g., RADOS objects) are the basic storage units in the underlying object storage layer used to actually persist file data blocks; each data object is addressed by a unique object identifier. Object extents (e.g., extents) refer to the storage segment of accessed data within a specific data object, usually characterized by the relative starting offset and segment length within that data object.

[0047] Specifically, upon receiving an access request for a target file, the mapping component within the client (such as the Striper component) extracts parameters such as the data block size and object size from the file layout information of the target file. Based on these parameters, it performs block and mapping calculations on the file offset range (i.e., the starting offset and read length within the file) carried by the access request. First, through logical operations such as division and modulo, it calculates which underlying data objects the file offset range spans, thereby obtaining the specific object number to be accessed. Then, it further calculates the relative starting position and the corresponding data slice length within each hit data object, thus splitting and converting continuous file access requests into object ranges targeting at least one underlying data object, completing the structured transformation from upper-layer file system semantics to lower-layer object storage semantics.

[0048] In step S130, if the data corresponding to the object range is loaded into the cache entry of the local memory cache, or if the cache entry of the data corresponding to the object range is hit in the local memory cache, a cache protection flag is set for the cache entry according to the effective caching strategy.

[0049] Specifically, when the client actually reads file data, there are two scenarios: cache miss and cache hit. If the required data is not yet found locally (i.e., cache miss), the client requests the data corresponding to the object scope from the backend storage and loads it into its own locally maintained memory cache, generating a corresponding cache entry to enable low-latency reading on subsequent repeated accesses. If the required data is already found in the local memory cache, the existing cache entry corresponding to that data is directly determined. This local memory cache can be an object cache directly managed by the client component, such as ObjectCacher in the user-space client, or it can be memory managed by the operating system's page cache adaptation layer. The data in the cache exists in the form of cache entries (such as buffer data blocks, object descriptor structures, or page cache descriptors).

[0050] While data is being loaded and corresponding cache entries are being generated, or if a corresponding cache entry is hit in the local memory cache and it is detected that the cache entry has not yet been marked with a cache protection flag (e.g., because the previous access occurred before the new cache policy was issued), the client checks the valid cache policy corresponding to the previously obtained file. If the valid cache policy indicates that the client's data needs to be cached persistently, the client records the corresponding cache protection flag (e.g., client_cache_pin) in the internal data structure of the cache entry. This indicates that the cached data belongs to a protected critical directory and cannot be arbitrarily evicted. This mechanism ensures that even if data is loaded into the cache before the policy is issued, it can still be promptly marked with a protection flag upon subsequent hits, thus preventing critical data from being mistakenly evicted due to the time difference in policy effectiveness.

[0051] Step S140: During the cache reclamation process of the local memory cache, skip or postpone the reclamation of cache entries with the cache protection flag set.

[0052] Specifically, as clients continuously read large amounts of data, the usage of local memory cache gradually increases. When the cache space is insufficient or reaches a set global threshold, the client's cache management module triggers a normal cache reclamation (or cache eviction) process to free up space for new data. In traditional eviction mechanisms (such as the ordinary LRU strategy), data in critical directories is easily squeezed out by other large amounts of unrelated ordinary data.

[0053] In this embodiment of the invention, when the cache reclamation module traverses the candidate cache entries for eviction (or pending reclamation), it actively identifies whether each cache entry carries a cache protection tag. For cache entries with cache protection tags, the cache reclamation module will protect them, for example, by skipping the reclamation of the cache entry in this cache reclamation process, or by moving it to a special protected queue to postpone its reclamation, while prioritizing the eviction of ordinary cache entries that do not have cache protection tags.

[0054] The method provided in this invention allows the client to obtain an effective caching strategy for target files in a target directory from a metadata server, transforming traditional global indiscriminate caching control into fine-grained directory-level caching control. When processing actual file access requests, the client uses file layout information to map the file offset range to the object range of the underlying data object. When data is loaded into the local memory cache or when the corresponding data is hit in the local memory cache, a cache protection flag is set for the corresponding cache entry according to the effective caching strategy. This mechanism enables the client to identify protected critical file data when performing cache reclamation operations on the local memory cache, and thus skip or postpone the reclamation of these cache entries with cache protection flags. This effectively avoids data in critical directories being squeezed out of the local memory cache due to a large number of irrelevant data reading operations in the system, thereby reducing the need for repeated cross-network remote reading of critical data and reducing access latency jitter.

[0055] Based on the above embodiments, step S110 specifically includes: Step S111: When performing any one of the following operations on the target file: path resolution, opening, or first reading, or when receiving a cache permission grant message or client session message for the target file, receive the effective caching policy issued by the metadata server. Step S112: Record the effective caching strategy in the inode or file handle corresponding to the target file. The effective caching strategy includes at least one of the following: strategy identifier, strategy version, cache priority, and capacity limit.

[0056] Specifically, in actual interactions within a distributed file system, before reading or writing file data, the client needs to interact with the metadata server to obtain the file's metadata. Simultaneously, the metadata server manages the client's file access permissions and session state through a specific asynchronous messaging mechanism. This embodiment of the invention utilizes these basic interaction processes and existing message channels. When the client initiates any of the following operations to the metadata server: path resolution, open, or first read operation for a target file; or when the client receives a cache permission grant message for the target file from the metadata server (e.g., a cap grant message in CephFS) or a client session message, the metadata server, while returning basic metadata (such as file size, permissions, file layout, etc.) or sending the aforementioned management message, simultaneously sends the effective caching policy configured for the directory to which the target file belongs. This eliminates the need for the client to initiate additional network requests to obtain the caching policy.

[0057] After receiving a valid caching policy from the metadata server, the client will record the valid caching policy in the inode (an object representing file metadata in the file system) or file handle of the target file maintained in the client's local memory in order to ensure that subsequent high-frequency data read and write requests can quickly obtain policy control information.

[0058] The recorded effective caching policy can include one or more policy information such as policy identifier, policy version, cache priority, and capacity limit to achieve fine-grained control. Here, the policy identifier (represented as `policy_id`) is a unique identifier associated with specific policy content. The policy version (represented as `policy_epoch`) is the specific version number of the effective caching policy. When a directory caching policy on the metadata server side is added, modified, or deleted, this version number can be used to notify and trigger the client to refresh the locally recorded effective caching policy. The cache priority (represented as `priority`) is used to determine the order in which multiple protected directories are evicted when the system faces memory pressure. Its configurable values ​​include `low`, `normal`, `high`, and `critical`, and can also be configured as a specific numerical priority, such as 0 to 100, with higher values ​​indicating later evicting. Capacity limits include reserved cache capacity (represented as reserve_bytes) and maximum cache capacity (represented as max_bytes). Reserved cache capacity refers to the target cache capacity reserved for this directory. It can be configured as a non-negative integer in bytes, and also supports capacity units such as MiB and GiB. When the configuration value is 0, it means that no explicit reserved capacity is set, and it is only processed according to priority or maximum capacity constraints. Maximum cache capacity refers to the maximum cache capacity allowed for this directory. It can be configured as a non-negative integer in bytes, and also supports capacity units such as MiB and GiB. When the configuration value is 0, it means that no directory-level maximum capacity is set, but it is still subject to the client or server global cache limit.

[0059] The method provided in this invention reuses existing metadata interaction channels and asynchronous message mechanisms to issue directory-level caching policies during file path resolution, open operation, first read operation, and receiving messages such as cache permission granting or client session, and records them in local inodes or file handles. This avoids the client from initiating independent network requests to obtain caching policies, reducing communication overhead, and ensures that subsequent read and write operations can achieve low-latency policy fast query.

[0060] Based on any of the above embodiments, step S130 specifically includes: Step S131: Determine the file to which the cache entry belongs and the effective caching strategy corresponding to the file based on the object range; Step S132: Record the protection identifier, the policy identifier of the effective caching policy, and the directory identifier of the target directory in the description structure of the cache entry, wherein the cache entry includes buffered data entries, object entries, or page cache entries in the object cache.

[0061] Specifically, when the client reads and writes file data, internal client components (such as the Striper component) not only need to convert from file offset range to object range, but also need to maintain contextual relationships in the data flow chain. When the underlying data is returned and ready to be written to the local memory cache, the client determines the specific file to which the currently pending cache entry (i.e., the data block retrieved from the remote end) belongs by looking up the mapping relationship based on the object range. Then, based on the file's inode or file handle, it reads the effective caching policy corresponding to the file, thereby confirming whether the newly loaded cache entry needs to be protected by the cache.

[0062] When protection is deemed necessary, the client marks the cache entry in its local memory cache (i.e., sets a cache protection flag). Specifically, this is achieved by adding and recording corresponding control fields directly into the cache entry's description structure. These fields primarily include a protection flag, a policy flag for the effective caching strategy, and a directory flag for the target directory. The protection flag (represented as `client_cache_pin`) can be a boolean type, serving as the most intuitive switch for the cache eviction module to determine whether to skip eviction. The policy flag (`policy_id`) is used to associate the priority and capacity limit of the corresponding caching strategy. The directory flag (represented as `owner_dir_ino`) is used to accurately count the total number of bytes of protected memory currently occupied by the directory.

[0063] Furthermore, depending on the underlying caching architecture of the client, the aforementioned tagged cache entries can be buffered data entries (i.e., BufferHead) under the user-space object cache management architecture or higher-level object entries (i.e., Object), or operating system page cache entries in the kernel CephFS client, or page cache entries maintained by the operating system in the FUSE scenario.

[0064] Based on any of the above embodiments, step S140 specifically includes: Step S141: If the local memory cache occupancy exceeds a preset threshold, candidate cache entries to be reclaimed are determined from the local memory cache according to a preset cache eviction policy. Step S142: If the candidate cache entry has the cache protection flag set, skip the recycling of the candidate cache entry, or migrate the candidate cache entry to the protected cache queue; Step S143: If the candidate cache entry does not have the cache protection flag set, the candidate cache entry is reclaimed.

[0065] Specifically, during client operation, when the continuous reading of new data causes the total usage of the local memory cache to exceed a preset threshold, the system will trigger a normal cache reclamation mechanism. At this stage, the cache reclamation module will first scan and determine the candidate cache entries to be reclaimed from the normal cache list according to the preset cache eviction policy (such as the global LRU policy).

[0066] In traditional processes, candidate cache entries are destroyed and released as soon as they are selected. However, in this embodiment of the invention, the cache reclamation module adds a protection check before actual release: it checks whether the candidate cache entry's description structure contains a cache protection flag (e.g., whether `client_cache_pin` is true). If the flag is present, it indicates that the data belongs to a critical directory requiring protection. In this case, the system will either skip reclamation of the candidate cache entry (keeping it in the original memory cache) or migrate it to a specially maintained protected cache queue, thus physically isolating protected data from ordinary data and preventing accidental destruction.

[0067] If, upon inspection, it is found that the candidate cache entry does not have a cache protection flag set, it indicates that the data block stored in the candidate cache entry belongs to ordinary directory data without a configured cache resident policy. For such cache entries, the cache reclamation module will directly reclaim and release the memory space occupied by them according to the normal memory release process, so that it can be used by new read and write requests.

[0068] The method provided in this embodiment of the invention introduces lightweight marking and judgment logic into the cache reclamation process, which not only retains the normal reclamation of ordinary data by the global eviction policy, but also achieves the interception and isolation of critical directory data with a small performance overhead, avoiding the accidental destruction of critical directory data and improving the execution efficiency of the cache reclamation process.

[0069] Based on any of the above embodiments, the effective caching strategy includes caching modes, which include a persistent mode and a priority mode.

[0070] Specifically, to accommodate the varying trade-offs between data protection and memory resource utilization in different business scenarios, administrators can configure a caching mode representing the strength of protection in the persistent caching policy for the target directory. When a client obtains the effective caching policy for the target files in that target directory, it will also receive that caching mode information.

[0071] Here, the configuration values ​​for cache mode can include `resident` and `priority`. `resident` refers to the resident mode, representing the highest level of protection. It instructs the cache eviction process to skip the corresponding cache entry, keeping it as resident in the memory cache as possible. Degradation is only allowed when degradation conditions are triggered (such as exceeding the directory's maximum capacity, insufficient client memory, forced release by the administrator, or data consistency failure). `priority` refers to the priority mode, indicating that the corresponding cache entry can still participate in the cache eviction process, but its priority is increased to ensure its eviction order is later than ordinary cache entries without cache protection flags.

[0072] Furthermore, step S140 specifically includes: If the local memory cache is insufficient, cache reclamation is performed according to the directory-aware hierarchical order. The directory-aware hierarchical order is as follows: first, cache entries that have not been marked with the cache protection flag are reclaimed; if the target release capacity has not been reached, cache entries with the cache mode set to the priority mode are reclaimed; if the target release capacity has not been reached and the degradation conditions are met, cache entries with the cache mode set to the resident mode are reclaimed.

[0073] Specifically, during prolonged system operation or sudden large-scale read / write operations, the client's local memory cache is prone to reaching its capacity limit, leading to insufficient cache space. In this situation, if the system is configured with a directory-aware cache eviction policy, the cache reclamation module will no longer mechanically perform blind reclamation based on the global LRU policy. Instead, it will initiate a new mechanism that executes cache reclamation according to a directory-aware hierarchical order. This mechanism divides cache entries into different tiers based on their protection strength and releases space tier by tier based on the target release capacity.

[0074] When performing tiered cache reclamation, the cache reclamation module first reclaims cache entries without cache protection flags, i.e., data blocks in ordinary directories where cache protection policies are not enabled. After scanning and reclaiming ordinary cache entries, the system determines whether the currently released cache space has reached the expected target release capacity. If the target release capacity has not been reached, the system will reclaim cache entries in priority mode, because although this data is important, the policy allows it to be evicted when memory cache pressure is high. Finally, after reclaiming priority mode cache entries, if the target release capacity has still not been reached, and if the degradation conditions are met (e.g., the system is about to crash due to a severe memory alarm, the administrator issues a forced release command, or the underlying data has experienced consistency failure), cache entries in resident mode will be reclaimed.

[0075] The method provided in this invention introduces a directory-aware hierarchical ordering mechanism and an on-demand reclamation logic based on target release capacity. This ensures that ordinary cache entries and priority mode cache entries can be released in an orderly manner under normal pressure to maintain the basic operation of the system, while also ensuring the safety of resident mode cache entries under non-extreme conditions. This achieves the integration of resident protection of local critical data and global memory cache health management in complex business scenarios.

[0076] Based on any of the above embodiments, the effective caching strategy further includes the maximum cache capacity of the target directory. Accordingly, after step S130, the method further includes: If the maximum cache capacity is non-zero and the protected cache capacity of the target directory exceeds the maximum cache capacity, the protected cache entry corresponding to the target directory is reclaimed according to at least one of access time, file priority, or object offset within the protected cache entry corresponding to the target directory.

[0077] Specifically, when configuring a persistent caching policy for a target directory, in addition to setting the caching mode, administrators typically also set a capacity constraint field. This means the effective caching policy also includes the maximum cache capacity of the target directory (i.e., max_bytes). After the client completes data loading and sets cache protection flags for cache entries, the client will continuously accumulate the total number of protected cache bytes currently occupied by the target directory (i.e., the protected cache capacity).

[0078] When the system detects that the maximum cache capacity is non-zero and the protected cache capacity of the target directory exceeds the maximum cache capacity, the system will not preempt or reclaim cache from other directories. Instead, it will trigger an internal eviction mechanism. Specifically, the system directly selects cache entries to be reclaimed from the protected cache entries corresponding to the target directory based on access time, file priority, or object offset, until the cache usage of that directory falls back below the maximum cache capacity. Here, access time refers to performing a local LRU within the directory, evicting the least accessed cache entry in that directory; file priority refers to evicting lower-priority subfile data in that directory; and object offset refers to evicting cache entries corresponding to previously read object data in sequential read scenarios.

[0079] The method provided in this invention, by introducing a hard constraint on the maximum cache capacity of the protected directory and a local eviction mechanism within the directory, can protect critical data while avoiding a single directory from occupying client memory indefinitely, thereby improving storage stability in multi-task concurrent scenarios.

[0080] Based on any of the above embodiments, under specific memory pressure or abnormal scenarios, in order to ensure the operation of the entire system and the correctness of data logic, the protected cache must be allowed to reduce its protection strength. Therefore, the method further includes: If the degradation condition is met, the cache protection flag of the cache entry is removed, or the cache mode of the cache entry is adjusted from the resident mode to the priority mode; The downgrade condition is any one of the following: the maximum cache capacity is a non-zero value and the protected cache capacity corresponding to the target directory exceeds the maximum cache capacity; the available memory of the client is lower than the security threshold; a forced release instruction is received; the cache resident policy of the target directory is deleted or disabled; the data of the target file is modified, truncated, deleted, or its consistency fails; or the policy version of the local record is determined to be expired after reconnecting with the metadata server.

[0081] Specifically, when the system detects an anomaly that triggers a degradation, the client will proactively adjust these cache entries that originally enjoyed cache resident protection. The specific degradation actions include two types: one is to remove the cache protection flag of the cache entry (e.g., resetting client_cache_pin to false), making it equivalent to a normal cache entry, thus allowing it to be evicted by the regular LRU algorithm; the other is to adjust the cache mode of the cache entry from the strongest resident mode to a priority mode, allowing it to be reclaimed in stages when memory cache is extremely scarce.

[0082] The various degradation conditions that trigger the above degradation actions cover multiple dimensions, including capacity, system security, manual intervention, and consistency. Specifically, when the maximum cache capacity is non-zero and the protected cache capacity corresponding to the target directory exceeds the maximum cache capacity (i.e., exceeds the max_bytes limit), degradation is performed to limit its unlimited expansion; when the available memory of the client is lower than the safety threshold, the protected memory cache needs to be relinquished to prevent the operating system from running out of memory and crashing; when the administrator actively issues and the system receives a forced release command for maintenance purposes, degradation is required; when the administrator adjusts the cache resident policy of the target directory to delete or disable on the metadata server side, degradation is triggered; when the data of the target file is modified, truncated, deleted, or its consistency fails, it indicates that the data in the cache is already dirty data, and continuing to protect it is meaningless, so degradation is triggered; when the client determines that the policy version recorded locally has expired after experiencing a network interruption and reconnecting to the metadata server, degradation is required to wait for the latest policy to be obtained.

[0083] The method provided in this invention avoids the caching resident strategy from crashing during a system memory crisis by establishing clear and multi-dimensional degradation conditions and corresponding degradation actions, while also preventing invalid policies or dirty data from occupying memory resources for a long time.

[0084] Based on any of the above embodiments, after step S130, the method further includes: If a truncation event, deletion event, renaming event, cache permission revocation event, or snapshot rollback event is detected for the target file, the corresponding cache entries are cleaned up or the cache protection flag of the corresponding cache entries is removed, based on the inode corresponding to the target file and the object scope.

[0085] Specifically, during routine read and write access to file data by clients, several file system events may occur that disrupt the continuity or validity of file data. These include truncate events, unlink events, rename events, cap revoke events, and snapshot rollback events. Here, a truncate event refers to a file being shortened; a unlink event refers to a file being permanently deleted; a rename event refers to a file being moved out of a protected directory; a cap revoke event refers to the metadata server reclaiming the current client's caching capabilities due to concurrent write conflicts from multiple clients; and a snapshot rollback event refers to the file system performing an operation to restore the file to a historical state.

[0086] If any of the above events targeting the target file are detected, the data that the client originally stored in the memory cache with cache protection markings has become invalid or no longer has a legitimate reason to be protected. At this time, the client will search and locate the invalid data blocks in the memory pool based on the inode corresponding to the target file and the previously mapped object range, thereby directly cleaning up the corresponding cache entries or removing the cache protection markings of the corresponding cache entries, so that they are evicted during the normal cache reclamation process.

[0087] The method provided in this invention, by coordinating with the native metadata change and consistency maintenance events of the file system, ensures that the data state in the client's local protected cache is always strongly consistent with the actual data state in the backend, thereby avoiding errors caused by subsequent business reading old data.

[0088] Based on any of the above embodiments, after step S120, the method further includes: When the effective caching policy indicates that server-side cache protection is enabled, send an object read / write request carrying an object-level cache protection tag to the object storage service provider, or send an independent object protection request to the object storage service provider. The object-level cache protection tag or the object protection request is determined based on the object scope and the effective caching policy. It is used to instruct the object storage service to set a server-side cache protection tag for the server-side cache entries corresponding to the object scope, and to skip or delay the recycling of server-side cache entries with the server-side cache protection tag during the server-side cache recycling process.

[0089] Specifically, when configuring a persistent caching policy for a target directory, administrators can protect client-side caches and enable server-side cache protection individually or simultaneously. If an effective caching policy indicates that server-side cache protection is enabled, after the client completes the mapping from file offset ranges to data objects, it will initiate an object read / write request to the backend object storage service (OSD) and attach an object-level cache protection tag to the request protocol packet; or, in some scenarios where immediate read / write is not required but protection policies need to be pre-deployed or asynchronously distributed, a separate object protection request can be sent directly to the object storage service.

[0090] Since the backend object storage service only perceives the underlying RADOS data objects and lacks the directory semantics of higher-level file systems like CephFS, it relies on the client to pass the mapping results from directory policies to objects. The object-level cache protection label or independent object protection request here is the key control information determined and extracted by the client based on the object scope and effective caching policies, and may include policy identifiers, object priorities, etc.

[0091] When an OSD receives an object read / write request or a standalone object protection request with an object-level cache protection tag, the tag or request instructs it to proactively set a server-side cache protection flag (e.g., `server_cache_pin`) for the corresponding server-side cache entry when loading object data into the server-side local cache (such as BlueStore cache). With this flag, the OSD can also identify, skip, or postpone the reclamation of server-side cache entries with the `server_cache_pin` flag when performing server-side cache reclamation due to its own memory pressure.

[0092] The method provided in this invention allows the client to carry a cache protection tag in the object read / write request, or to flexibly send an independent object protection request, so as to successfully distribute the directory-level cache resident policy to the server, thereby reducing the probability of cache eviction on the critical data server and ensuring the overall read / write performance of the cluster.

[0093] Based on any of the above embodiments Figure 2 This is the second flowchart of the directory-level data cache management method provided by the present invention, as follows: Figure 2As shown, this method is applied to a metadata server of a distributed file system, and the method includes: Step S210: Receive a policy setting request for the target directory, and obtain the cache persistence policy configured for the target directory from the policy setting request; Step S220: Determine the effective caching strategy for the target files in the target directory based on the caching resident strategy; Step S230: The effective caching policy for the target file is sent to the client, so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to an object range of at least one data object based on the file layout information of the target file. If the data corresponding to the object range is loaded into a cache entry in the client's local memory cache, or if a cache entry for the data corresponding to the object range is hit in the local memory cache, the client sets a cache protection flag for the cache entry according to the effective caching policy. The client is used to skip or postpone the reclamation of cache entries with the cache protection flag during the cache reclamation process of the local memory cache.

[0094] It should be noted that the execution entity of the method provided in this embodiment of the invention can be a metadata server (MDS) of a distributed file system. This embodiment of the invention mainly describes how the MDS manages and distributes directory-level data caching strategies.

[0095] Specifically, in a distributed file system, administrators or automated operation and maintenance programs issue configuration instructions for a specific directory (i.e., the target directory) through extended attributes, management commands, or API interfaces. The metadata server receives policy setting requests for the target directory in real time. By parsing the request, it can obtain the persistent caching policy specifically configured by the administrator for the target directory.

[0096] After obtaining the directory-level cache persistence policy, the metadata server maintains this policy internally. When a client subsequently initiates an access to a specific file (i.e., the target file) under the target directory, the metadata server determines the effective cache policy for that target file based on the cache persistence policy. Here, the process of determining the effective cache policy may involve determining whether the file directly inherits the policy configuration of the parent directory, thus forming an accurate policy mapping for a single file.

[0097] After determining an effective caching strategy, the metadata server proactively sends the effective caching strategy for the target file to the client through communication interactions (such as responding to client path resolution operations, opening file operations, or granting caching permissions). Upon receiving the effective caching strategy, the client records it in the inode or file handle corresponding to the target file so that the effective caching strategy can be applied subsequently.

[0098] Specifically, when the client subsequently receives an access request for the target file, it maps the file offset range indicated by the access request to the object range of at least one data object based on the file layout information of the target file. Then, if the data corresponding to the object range is loaded into a cache entry in the client's local memory cache, or if a cache entry for the corresponding data is hit in the local memory cache and it is detected that the cache entry has not yet been marked with a cache protection flag (e.g., because the previous access occurred before the new caching policy was issued), the client can set a cache protection flag for the cache entry according to the previously received and recorded valid caching policy. Finally, when the client performs cache reclamation on the local memory cache due to insufficient cache space, it can use this flag to skip or postpone the reclamation of cache entries with cache protection flags.

[0099] The method provided in this invention constructs a globally unified data caching strategy expression and distribution management in a distributed file system by centrally receiving, maintaining and distributing directory-level caching persistence strategies on the metadata server side, which can coordinate the data caching behavior of multiple clients on the same directory.

[0100] Based on any of the above embodiments, after step S210, the method further includes: The validity of the policy fields in the cache persistence policy is verified. The policy fields include at least one of the following: server-side cache protection enabled status, client-side cache protection enabled status, reserved cache capacity, maximum cache capacity, cache priority, subdirectory inheritance identifier, cache mode, and cache eviction policy. The cache persistence policy that has passed the verification is persisted and a policy version corresponding to the cache persistence policy is generated.

[0101] Specifically, because cached resources are valuable, the metadata server cannot blindly accept all configurations issued by the administrator and needs to intercept erroneous or unreasonable parameters at this stage. Therefore, the metadata server will verify the legality of the execution format and logic of the various policy fields mentioned above in the cache persistence policy.

[0102] The server-side cache protection enable status (represented by `server_pin`) indicates whether server-side cache protection is enabled. Its configurable value is 0, 1, or `false` or `true`, with a default value of 0. When `server_pin` is enabled, the system can mark the RADOS object cache corresponding to the directory data as `server_cache_pin`, causing these object caches to be skipped or delayed during OSD or server-side cache reclamation. The client-side cache protection enable status (represented by `client_pin`) indicates whether client-side cache protection is enabled. Its configurable value is 0, 1, or `false` or `true`, with a default value of 0. When `client_pin` is enabled, the system can mark the client-side local cache entries corresponding to the directory data as `client_cache_pin`, causing these local cache entries to be skipped or delayed during client-side cache or page cache reclamation. It should be understood that `server_pin` and `client_pin` can be enabled individually or simultaneously. For example, when only server_pin is enabled, it mainly protects the server-side object cache, which is suitable for scenarios where multiple clients share the same batch of hot objects; when only client_pin is enabled, it mainly protects the local memory cache of the accessing client, which is suitable for scenarios that are sensitive to the latency of repeated access by a single client; when both are enabled, the directory data has protection semantics in both the server-side cache and the client-side cache.

[0103] Reserved cache capacity (i.e., `reserve_bytes`) refers to the target cache capacity reserved for the directory. Its configurable value is a non-negative integer in bytes, and also supports MiB, GiB, and other capacity units. A value of 0 indicates that no explicit reserved capacity is set, and processing is based solely on priority or maximum capacity constraints. Maximum cache capacity (i.e., `max_bytes`) refers to the maximum cache capacity allowed for this directory. Its configurable value is a non-negative integer in bytes, and also supports MiB, GiB, and other capacity units. A value of 0 indicates that no directory-level maximum capacity is set, but it is still subject to the client or server-side global cache limit.

[0104] Cache priority (denoted as priority) can be used to sort the protection strength among multiple directories. Its configurable values ​​include low, normal, high, and critical, and it can also be implemented as a numerical priority, such as 0 to 100, with larger values ​​indicating later eviction. The subdirectory inheritance flag (inherit) indicates whether the caching policy is inherited by subdirectories and files. Its configurable values ​​are 0, 1, false, and true, with the default value being true. When configured as false, the caching policy only applies to files directly associated with the current directory or to a scope defined by the implementation.

[0105] The caching mode (denoted as `mode`) describes the strength of cache protection for this directory. Its configurable values ​​include `resident` and `priority`, where `resident` indicates resident mode and `priority` indicates priority mode. The cache eviction policy (denoted as `eviction_policy`) refers to the cache eviction policy for the directory. Its configurable values ​​include `default`, `directory_tiered`, `quota_first`, `priority_weighted`, and `strict_resident`. This policy field will be explained in detail in later examples and will not be elaborated upon here.

[0106] Specifically, when validating the above policy fields, for the server-side cache protection enabled status and client-side cache protection enabled status, the main checks are whether they are valid Boolean values ​​or 0 or 1 values; for the reserved cache capacity and maximum cache capacity, the main checks are whether they are non-negative integers or supported units, and if the maximum cache capacity is non-zero, the checks are whether the reserved cache capacity is less than or equal to the maximum cache capacity; if the maximum cache capacity is zero, no size check is performed between the reserved cache capacity and the maximum cache capacity; for cache priority, the checks are whether it belongs to a predefined enumeration value (such as low, normal, high, critical) or is within a reasonable numerical range (such as 0 to 100); for the subdirectory inheritance identifier, the checks are whether it is a valid Boolean value, used to determine whether the policy only applies to the current directory or automatically overwrites all its subdirectories and files; for cache mode, the checks are whether it is a valid mode option, such as resident mode or priority mode; for cache eviction policy, the checks are whether it is a system-supported eviction algorithm name.

[0107] If all the above fields are validated, the metadata server will persist the validated cache persistence policy, for example, by saving it to the backend metadata storage pool, to ensure that the protection configuration of these directories will not be lost even if the metadata server restarts. Simultaneously, the metadata server will generate a unique, incrementing policy version for this successful policy change. This version number is a crucial benchmark for subsequently notifying each client whether its locally recorded policy has expired.

[0108] The method provided in this invention avoids cache capacity overflow or system logic anomalies caused by human configuration errors by introducing pre-emptive policy validity verification in the metadata server. At the same time, persistence and version number generation provide a metadata management foundation for asynchronous policy distribution and consistent refresh among multiple clients in a distributed environment.

[0109] Based on any of the above embodiments, the cache eviction strategy is any one of the following: The default policy is used to instruct the global cache eviction policy (also known as the cache removal policy) to be used for cache eviction. The directory-aware hierarchical eviction policy is used to indicate the hierarchical eviction of cache entries in the following order: cache entries without cache protection flags, cache entries in priority mode, and cache entries in resident mode. The capacity-first strategy is used to indicate that if the cache capacity occupied by a directory exceeds the maximum cache capacity, cache reclamation should be performed within that directory first. Priority-weighted strategy is used to instruct the calculation of eviction weights based on cache priority and the selection of cache entries to be reclaimed; A strict resident policy is used to instruct that cache entries in resident mode be reclaimed only if the mandatory degradation conditions are met.

[0110] Specifically, if configured with the default policy, the underlying system will not change the original global cache reclamation behavior. Even if cache protection is enabled, the system only uses these cache protection flags to perform memory usage statistics or observations, providing data reference for subsequent policy switching, but does not require the cache reclamation module to change the eviction order.

[0111] If configured with a directory-aware tiered eviction policy (i.e., directory_tiered), regular cache entries will be evicted first, followed by cache entries in priority mode, and finally, cache entries in resident mode will only be evicted when capacity is exceeded or when degradation conditions are triggered.

[0112] The capacity-first strategy (i.e., quota_first) emphasizes the fairness of resource quotas and does not allow any protected directory to occupy more than its due share. Once the limit is exceeded, it is required to self-eliminate internally, rather than expanding outward to squeeze the cache space of other ordinary directories or protected directories.

[0113] The priority-weighted strategy (i.e., priority_weighted) transforms cache priority into eviction weight. The higher the priority of a directory, the lower the probability that its data will be selected for eviction during scanning. It is suitable for complex business scenarios where a large number of directories of different importance coexist.

[0114] If configured with a strict resident policy (i.e., strict_resident), it means that cache entries corresponding to the resident mode will not be evicted as much as possible, and will only be released under forced degradation conditions.

[0115] It is understood that, by introducing a cache eviction strategy, the embodiments of the present invention enable the server-side and client-side cache reclamation to no longer rely solely on global LRU or global capacity thresholds for reclamation, but instead combine directory caching strategies, cache protection flags, directory capacity, and cache modes for hierarchical processing. This strategy can be applied to client-side ObjectCacher, client-side page cache adaptation layer, OSD object cache, or BlueStore cache related reclamation paths.

[0116] Taking the directory_tiered strategy as an example, Figure 3 This is a flowchart of the directory-aware hierarchical elimination strategy provided by the present invention, as follows: Figure 3 As shown, the processing flow specifically includes the following steps: When the client or server detects insufficient cache space, the cache reclamation module first initiates the first round of eviction, which reclaims ordinary cache (cache entries without cache protection flags). After completing the first round of reclamation, the system determines whether enough space has been released. If yes, the reclamation ends directly; if no (i.e., space is still insufficient), the second round of eviction begins, reclaiming priority mode directory cache, i.e., priority mode cache entries. After completing the second round of reclamation, the system again determines whether enough space has been released. If yes, the reclamation ends; if no, the most critical protected resident data is affected. At this point, the system checks whether the resident mode directory cache exceeds max_bytes. If yes (i.e., resident data exceeds the limit), only the excess portion or the portion meeting the degradation conditions (corresponding to resident mode cache entries) is reclaimed; if no, since there is no more to evict, the system triggers memory pressure degradation or administrator policies, forcibly removing some protection flags.

[0117] Based on any of the above embodiments, after the step of persistently storing the verified cache resident policy, the method further includes: Maintain policy status statistics information corresponding to the cache resident policy. The policy status statistics information includes at least one of the following: client information holding the cache resident policy, protected cache capacity reported by each client, and protected cache capacity reported by the object storage service client.

[0118] Specifically, in a distributed, multi-client access environment, simply issuing policies without subsequent tracking can easily lead to the entire cluster's cache resources being occupied by some hot directories. Therefore, after persisting the policies, the Metadata Server (MDS) continuously maintains the policy status statistics corresponding to the cached persistent policies.

[0119] Specifically, MDS records which clients are currently accessing the directory, i.e., the client information holding the cache resident policy, and receives status reports from the underlying layer. It then summarizes and calculates the protected cache capacity reported by each client (i.e., the total size of the client_cache_pin actually effective on each client's local machine) and the protected cache capacity reported by the backend object storage service (i.e., the total size of the server_cache_pin effective on the OSD side). Through these global statistical data, the system can accurately determine whether a certain policy exceeds the globally recommended cache capacity.

[0120] Based on any of the above embodiments, step S220 specifically includes: If the target file does not have a caching policy configured, search upwards through the parent directory of the target file for the directory with the most recently configured caching resident policy, and determine the caching resident policy configured in the found directory as the effective caching policy for the target file.

[0121] Specifically, in a large file system, requiring administrators to configure a caching persistence policy for each file individually would not only incur extremely high management costs but also result in significant metadata storage overhead. Therefore, this embodiment of the invention introduces a policy inheritance mechanism at the directory tree level.

[0122] When a client requests access to a specific target file, the metadata server first checks whether the file itself is bound to a specific policy. If the target file does not have a caching policy configured, the metadata server uses the file system's hierarchical topology to search upwards along the target file's parent directories until it finds a directory that is most recently configured with a cache resident policy. The metadata server then uses the cache resident policy configured in the found directory as the final result to determine the effective caching policy for the target file and sends it to the client. If, during this process, the policy is configured so that subdirectories are not inherited (i.e., `inherit` is false), the server will skip that directory and continue searching upwards, or directly determine that there is no caching protection policy.

[0123] This invention simplifies the cache protection configuration in scenarios with a large number of files by introducing a policy inheritance mechanism that searches upwards along the directory tree. Administrators only need to declare the policy once at the top level or key level of the directory structure to achieve efficient and unified protection of all sub-file data in the directory. While ensuring the fineness of cache scheduling, it reduces the storage and computing overhead of the metadata server and improves the usability and management efficiency of the system.

[0124] Based on any of the above embodiments, the method further includes: If the cache persistence policy of the target directory is added, modified or deleted, update the policy version corresponding to the cache persistence policy and determine the list of clients that currently hold the inode or cache permissions of the files in the target directory. A policy update notification is sent to each client in the client list so that each client updates its effective caching policy for local records.

[0125] Specifically, business needs are dynamic, and administrators may adjust data caching protection rules for specific directories at any time. When the metadata server detects that the caching policy for a target directory has been dynamically added, modified (e.g., adjusted in terms of capacity quotas or priorities), or deleted (e.g., protection canceled) by the administrator, it immediately increments and updates the policy version on the backend. Subsequently, the metadata server retrieves its internally maintained client session and lock state information to determine the list of clients currently holding inode or caching permissions (capabilities, or caps) for files in the target directory. These clients are the stakeholders currently reading, writing, or caching data in that directory.

[0126] To prevent old policies from erroneously consuming memory or causing new policies to fail to take effect in a timely manner, the metadata server proactively sends policy update notifications to each client in the client list using existing communication channels (e.g., using CAP privileges to reclaim or update messages). This notification carries the latest policy version number, triggering each client to update the valid caching policy for its local records. Upon receiving the policy update notification, the client compares the policy version in its local records. If it finds an expired policy, it refreshes the policy parameters in its local inode or file handle, thereby triggering the recalculation or removal of the underlying cache entry protection flags.

[0127] This invention ensures the consistency and real-time performance of caching policies across the entire distributed cluster by establishing a proactive policy push mechanism based on version control and precise client location.

[0128] Based on any of the above embodiments Figure 4 This is a schematic diagram of the overall process of the directory-level data cache management method for distributed file systems provided by the present invention, as shown below. Figure 4 As shown, the execution flow of this method includes the following steps: Step S1: The administrator configures a persistent caching policy for the specified directory.

[0129] The protection mechanism of the entire system originates from the configuration distribution. First, the administrator or operation and maintenance system configures the cache resident policy for the key directories (such as AI model file directory, database index directory, etc.) that require low latency protection in business through extended attributes, API interfaces, etc., including cache mode, capacity limit, cache priority, etc.

[0130] Step S2: MDS verifies and persists the policy, processes policy integration, and sends the policy to the client.

[0131] Upon receiving the caching persistence policy, MDS verifies the validity of the policy fields and persists valid policies to the backend, while generating a new policy version number. Next, MDS internally processes the policy inheritance relationship within the directory tree, mapping directory-level policies to specific subfiles and distributing these policies to the client at appropriate times (such as when responding to client requests).

[0132] Step S3: When the client performs any one of the following operations: file path resolution, file open operation, or file first read operation, or when the client receives a message from the metadata server granting caching permissions for the target file or a client session message, the applicable directory caching strategy for the file is obtained and recorded.

[0133] Before actually reading or writing files, the client initiates metadata interaction operations such as file path resolution (lookup), opening, or initial read. Additionally, the client may receive management messages pushed by MDS via an asynchronous channel during runtime. During these stages, the client receives the directory caching policy (i.e., the effective caching policy) applicable to that specific file from MDS and caches it in local memory, specifically recording it in the inode or file handle corresponding to that file. This allows subsequent data read / write operations to quickly retrieve the policy with minimal overhead.

[0134] In step S4, when the client reads file data, it maps the file offset to an object range according to the layout.

[0135] When an upper-layer application initiates a real file read request, the request carries the logical offset within the file, i.e., the file offset range. The underlying mapping component on the client (such as Striper) reads the file's layout information, calculates and precisely segments the offset range based on the file dimension, and maps it to one or more object ranges (i.e., object number and object offset extent) that can be recognized by the underlying distributed object storage layer (RADOS).

[0136] In step S5, the client marks the read local cache entries with a cache protection tag.

[0137] After defining the scope of objects to be read, there are two scenarios: cache miss and cache hit. If the client finds that the data is missing in its local memory cache, it will fetch the data from the remote cache and load it locally (e.g., by generating a BufferHead or page cache entry). If it finds that the local memory cache already contains a cache entry for the data corresponding to the object scope, and that cache entry has not yet been marked with a cache protection flag, it will directly locate the existing cache entry. Simultaneously with the data being loaded into memory, or in the case of a local memory cache hit that has not yet been marked, the client will backtrack to the policy configuration of the file to which the data belongs. If it confirms that the data is protected by a resident policy, it will directly mark the cache protection flag (e.g., client_cache_pin) in the description structure of these local cache entries and accumulate the protected cache capacity at the directory level.

[0138] Step S6: The OSD receives the object request with the protection tag and adds the cache protection tag to the server.

[0139] In addition to client-side local protection, this embodiment of the invention also supports end-to-cloud collaboration. If server-side protection is enabled in the policy, the client will attach an object-level cache protection tag to the protocol packet when sending a read request to the OSD. When the OSD receives the request and loads the underlying data object into the server-side cache (such as BlueStore cache), it will also recognize the tag and mark the corresponding server-side cache entry with a server-side cache protection flag (i.e., server_cache_pin).

[0140] Step S7: When normal eviction is triggered, according to the cache mode and cache eviction policy in the directory caching policy, perform normal reclamation, delayed reclamation, or skip reclamation on cache entries with cache protection flags.

[0141] As the system continues to run, when the usage of either client-side local memory or server-side memory reaches a threshold requiring cache reclamation, the cache reclamation module will abandon the traditional global eviction policy and instead perform corresponding cache reclamation operations based on the cache mode and cache eviction policy in the directory caching policy (i.e., the effective caching policy). For example, when the cache eviction policy in the effective caching policy indicates the use of a directory-aware hierarchical eviction policy, the cache reclamation module will perform hierarchical reclamation in the order of normal cache, priority mode cache, and resident mode cache; when the cache eviction policy is the default policy, the cache reclamation module will use the original global cache eviction policy. Based on the determination results of the above policies, the system will ultimately perform corresponding normal reclamation, delayed reclamation, or skipped reclamation operations on cache entries with cache protection tags, thereby establishing the final retention state of the target data in memory.

[0142] The directory-level data cache management device provided by the present invention is described below. The directory-level data cache management device described below and the directory-level data cache management method described above can be referred to in correspondence.

[0143] Based on any of the above embodiments Figure 5 This is one of the structural schematic diagrams of the directory-level data cache management device provided by the present invention, such as... Figure 5 As shown, this device is used on a client of a distributed file system, and the device includes: The policy acquisition unit 510 is used to acquire the effective caching policy of the target file in the target directory from the metadata server. The effective caching policy is determined and issued by the metadata server according to the caching persistence policy configured for the target directory. The object mapping unit 520 is configured to, upon receiving an access request for the target file, map the file offset range indicated by the access request to the object range of at least one data object based on the file layout information of the target file. The cache marking unit 530 is used to set a cache protection mark for the cache entry according to the effective caching strategy when the data corresponding to the object range is loaded into the cache entry of the local memory cache, or when the cache entry of the data corresponding to the object range is hit in the local memory cache. The cache reclamation unit 540 is used to skip or postpone the reclamation of cache entries with the cache protection flag during the process of performing cache reclamation on the local memory cache.

[0144] The apparatus provided in this invention allows the client to obtain an effective caching strategy for target files in a target directory from a metadata server, transforming traditional global indiscriminate caching control into fine-grained directory-level caching control. When processing actual file access requests, the client uses file layout information to map the file offset range to the object range of the underlying data object. When data is loaded into the local memory cache or when the corresponding data is hit in the local memory cache, a cache protection flag is set for the corresponding cache entry according to the effective caching strategy. This mechanism enables the client to identify protected critical file data when performing cache reclamation operations on the local memory cache, thereby skipping or delaying the reclamation of these cache entries with cache protection flags. This effectively avoids data in critical directories being squeezed out of the local memory cache due to a large number of irrelevant data reading operations in the system, thereby reducing the need for repeated cross-network remote reading of critical data and reducing access latency jitter.

[0145] Based on any of the above embodiments, the strategy acquisition unit is specifically used for: If any one of the following operations is performed on the target file: path resolution, opening, or first read, or if a cache permission grant message or client session message for the target file is received, the effective caching policy issued by the metadata server shall be received. The effective caching strategy is recorded in the inode or file handle corresponding to the target file. The effective caching strategy includes at least one of the following: strategy identifier, strategy version, cache priority, and capacity limit.

[0146] Based on any of the above embodiments, the cache marking unit is specifically used for: Determine the file to which the cache entry belongs and the effective caching strategy corresponding to the file based on the object scope; The description structure of the cache entry records the protection identifier, the policy identifier of the effective caching policy, and the directory identifier of the target directory, wherein the cache entry includes buffered data entries, object entries, or page cache entries in the object cache.

[0147] Based on any of the above embodiments, the cache recycling unit is specifically used for: If the local memory cache usage exceeds a preset threshold, candidate cache entries to be reclaimed are determined from the local memory cache according to a preset cache eviction policy. If the candidate cache entry has the cache protection flag set, skip the recycling of the candidate cache entry, or migrate the candidate cache entry to the protected cache queue; If the candidate cache entry does not have the cache protection flag set, the candidate cache entry is reclaimed.

[0148] Based on any of the above embodiments, the effective caching strategy includes caching modes, which include a persistent mode and a priority mode; the cache reclamation unit is specifically used for: If the local memory cache is insufficient, cache reclamation is performed according to the directory-aware hierarchical order. The directory-aware hierarchical order is as follows: first, cache entries that have not been marked with the cache protection flag are reclaimed; if the target release capacity has not been reached, cache entries with the cache mode set to the priority mode are reclaimed; if the target release capacity has not been reached and the degradation conditions are met, cache entries with the cache mode set to the resident mode are reclaimed.

[0149] Based on any of the above embodiments, the effective caching strategy further includes the maximum cache capacity of the target directory. Correspondingly, the device further includes an internal recycling unit, which is used for: If the maximum cache capacity is non-zero and the protected cache capacity of the target directory exceeds the maximum cache capacity, the protected cache entry corresponding to the target directory is reclaimed according to at least one of access time, file priority, or object offset within the protected cache entry corresponding to the target directory.

[0150] Based on any of the above embodiments, the device further includes a degradation and recycling unit, the degradation and recycling unit being used for: If the degradation condition is met, the cache protection flag of the cache entry is removed, or the cache mode of the cache entry is adjusted from the resident mode to the priority mode; The downgrade condition is any one of the following: the maximum cache capacity is a non-zero value and the protected cache capacity corresponding to the target directory exceeds the maximum cache capacity; the available memory of the client is lower than the security threshold; a forced release instruction is received; the cache resident policy of the target directory is deleted or disabled; the data of the target file is modified, truncated, deleted, or its consistency fails; or the policy version of the local record is determined to be expired after reconnecting with the metadata server.

[0151] Based on any of the above embodiments, the device further includes a failure handling unit, the failure handling unit being used for: If a truncation event, deletion event, renaming event, cache permission revocation event, or snapshot rollback event is detected for the target file, the corresponding cache entries are cleaned up or the cache protection flag of the corresponding cache entries is removed, based on the inode corresponding to the target file and the object scope.

[0152] Based on any of the above embodiments, the device further includes a request sending unit, the request sending unit being used for: When the effective caching policy indicates that server-side cache protection is enabled, send an object read / write request carrying an object-level cache protection tag to the object storage service provider, or send an independent object protection request to the object storage service provider. The object-level cache protection tag or the object protection request is determined based on the object scope and the effective caching policy. It is used to instruct the object storage service to set a server-side cache protection tag for the server-side cache entries corresponding to the object scope, and to skip or delay the recycling of server-side cache entries with the server-side cache protection tag during the server-side cache recycling process.

[0153] Based on any of the above embodiments Figure 6 This is a second schematic diagram of the directory-level data cache management device provided by the present invention, as shown below. Figure 6 As shown, this device is used as a metadata server for a distributed file system, and the device includes: The request receiving unit 610 is used to receive a policy setting request for a target directory and obtain the cache resident policy configured for the target directory from the policy setting request. The strategy determination unit 620 is used to determine the effective caching strategy for the target files in the target directory based on the caching resident strategy. The policy distribution unit 630 is used to distribute an effective caching policy for the target file to the client, so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to the object range of at least one data object according to the file layout information of the target file, and sets a cache protection flag for the cache entry according to the effective caching policy when loading the data corresponding to the object range into the cache entry of the client's local memory cache, or when a cache entry for the data corresponding to the object range is hit in the local memory cache; the client is used to skip or postpone the reclamation of cache entries with the cache protection flag set during the cache reclamation process of the local memory cache.

[0154] Based on any of the above embodiments, the device further includes a policy verification unit, the policy verification unit being used for: The validity of the policy fields in the cache persistence policy is verified. The policy fields include at least one of the following: server-side cache protection enabled status, client-side cache protection enabled status, reserved cache capacity, maximum cache capacity, cache priority, subdirectory inheritance identifier, cache mode, and cache eviction policy. The cache persistence policy that has passed the verification is persisted and a policy version corresponding to the cache persistence policy is generated.

[0155] Based on any of the above embodiments, the device further includes a policy state maintenance unit, the policy state maintenance unit being used for: Maintain policy status statistics information corresponding to the cache resident policy. The policy status statistics information includes at least one of the following: client information holding the cache resident policy, protected cache capacity reported by each client, and protected cache capacity reported by the object storage service client.

[0156] Based on any of the above embodiments, the strategy determination unit is specifically used for: If the target file does not have a caching policy configured, search upwards through the parent directory of the target file for the directory with the most recently configured caching resident policy, and determine the caching resident policy configured in the found directory as the effective caching policy for the target file.

[0157] Based on any of the above embodiments, the device further includes a policy update unit, the policy update unit being used for: If the cache persistence policy of the target directory is added, modified or deleted, update the policy version corresponding to the cache persistence policy and determine the list of clients that currently hold the inode or cache permissions of the files in the target directory. A policy update notification is sent to each client in the client list so that each client updates its effective caching policy for local records.

[0158] This invention provides a client and a metadata server, both of which can be implemented in the form of computer devices. Figure 7 This is a schematic diagram of the structure of the computer device provided by the present invention, such as... Figure 7 As shown, the computer device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740.

[0159] In one embodiment, when Figure 7 When the computer device shown is a client, the processor 710 can call logical instructions in the memory 730 to execute a directory-level data cache management method. This method includes: obtaining an effective caching policy for target files in a target directory from a metadata server, wherein the effective caching policy is determined and issued by the metadata server based on a cache resident policy configured for the target directory; upon receiving an access request for the target file, mapping the file offset range indicated by the access request to an object range of at least one data object based on the file layout information of the target file; when loading data corresponding to the object range into a cache entry in the local memory cache, or when a cache entry for data corresponding to the object range is hit in the local memory cache, setting a cache protection flag for the cache entry according to the effective caching policy; and during cache reclamation of the local memory cache, skipping or delaying the reclamation of cache entries with the cache protection flag set.

[0160] In another embodiment, when Figure 7When the computer device shown is a metadata server, the processor 710 can call logical instructions in the memory 730 to execute a directory-level data cache management method. This method includes: receiving a policy setting request for a target directory and obtaining a cache resident policy configured for the target directory from the policy setting request; determining an effective cache policy for a target file under the target directory based on the cache resident policy; and issuing the effective cache policy for the target file to a client so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to an object range of at least one data object based on the file layout information of the target file, and sets a cache protection flag for the cache entry according to the effective cache policy when loading the data corresponding to the object range into a cache entry in the client's local memory cache, or when a cache entry for the data corresponding to the object range is hit in the local memory cache; the client is used to skip or postpone the reclamation of cache entries with the cache protection flag set during the cache reclamation process of the local memory cache.

[0161] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the directory-level data cache management method provided by the above methods. This method is applied to a client and includes: obtaining an effective caching strategy for a target file in a target directory from a metadata server, wherein the effective caching strategy is determined and issued by the metadata server according to a cache resident strategy configured for the target directory; upon receiving an access request for the target file, mapping the file offset range indicated by the access request to an object range of at least one data object according to the file layout information of the target file; when loading data corresponding to the object range into a cache entry in the local memory cache, or when a cache entry for data corresponding to the object range is hit in the local memory cache, setting a cache protection flag for the cache entry according to the effective caching strategy; and during the cache reclamation process of the local memory cache, skipping or delaying the reclamation of cache entries with the cache protection flag set.

[0163] When the computer program is executed by the processor, the computer can also execute the directory-level data cache management method provided by the above methods. This method is applied to a metadata server and includes: receiving a policy setting request for a target directory and obtaining a cache resident policy configured for the target directory from the policy setting request; determining an effective cache policy for a target file under the target directory according to the cache resident policy; and sending the effective cache policy for the target file to a client so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to an object range of at least one data object according to the file layout information of the target file, and sets a cache protection flag for the cache entry according to the effective cache policy when loading the data corresponding to the object range into the cache entry of the client's local memory cache, or when a cache entry for the data corresponding to the object range is hit in the local memory cache; the client is used to skip or postpone the reclamation of cache entries with the cache protection flag set during the cache reclamation process of the local memory cache.

[0164] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the directory-level data cache management method provided by the above methods. This method is applied to a client and includes: obtaining an effective caching policy for a target file in a target directory from a metadata server, the effective caching policy being determined and issued by the metadata server based on a cache resident policy configured for the target directory; upon receiving an access request for the target file, mapping the file offset range indicated by the access request to an object range of at least one data object based on the file layout information of the target file; when loading data corresponding to the object range into a cache entry in a local memory cache, or when a cache entry for data corresponding to the object range is hit in the local memory cache, setting a cache protection flag for the cache entry according to the effective caching policy; and during cache reclamation of the local memory cache, skipping or delaying the reclamation of cache entries with the cache protection flag set.

[0165] When executed by a processor, this computer program implements a directory-level data cache management method provided by the methods described above. This method is applied to a metadata server and includes: receiving a policy setting request for a target directory and obtaining a cache resident policy configured for the target directory from the policy setting request; determining an effective cache policy for a target file under the target directory based on the cache resident policy; and issuing the effective cache policy for the target file to a client so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to an object range of at least one data object based on the file layout information of the target file, and sets a cache protection flag for the cache entry according to the effective cache policy when loading the data corresponding to the object range into a cache entry in the client's local memory cache, or when a cache entry for the data corresponding to the object range is hit in the local memory cache; the client is configured to skip or postpone the reclamation of cache entries with the cache protection flag set during cache reclamation of the local memory cache.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A directory-level data cache management method, characterized in that, Clients used in distributed file systems include: Obtain the effective caching strategy for the target file in the target directory from the metadata server. The effective caching strategy is determined and issued by the metadata server based on the caching persistence strategy configured for the target directory. Upon receiving an access request for the target file, the file offset range indicated by the access request is mapped to the object range of at least one data object based on the file layout information of the target file. When the data corresponding to the object range is loaded into the cache entry of the local memory cache, or when the cache entry of the data corresponding to the object range is hit in the local memory cache, a cache protection flag is set for the cache entry according to the effective caching strategy; During the cache reclamation process of the local memory cache, cache entries with the cache protection flag set are skipped or the reclamation is delayed.

2. The directory-level data caching management method according to claim 1, characterized in that, The effective caching strategy for obtaining target files in the target directory from the metadata server includes: If any one of the following operations is performed on the target file: path resolution, opening, or first read, or if a cache permission grant message or client session message for the target file is received, the effective caching policy issued by the metadata server shall be received. The effective caching strategy is recorded in the inode or file handle corresponding to the target file. The effective caching strategy includes at least one of the following: strategy identifier, strategy version, cache priority, and capacity limit.

3. The directory-level data caching management method according to claim 1, characterized in that, The step of setting a cache protection flag for the cache entry according to the effective caching strategy includes: Determine the file to which the cache entry belongs and the effective caching strategy corresponding to the file based on the object scope; The description structure of the cache entry records the protection identifier, the policy identifier of the effective caching policy, and the directory identifier of the target directory, wherein the cache entry includes buffered data entries, object entries, or page cache entries in the object cache.

4. The directory-level data cache management method according to claim 1, characterized in that, The step of skipping or delaying the reclamation of cache entries marked with the cache protection flag during the cache reclamation process of the local memory cache includes: If the local memory cache usage exceeds a preset threshold, candidate cache entries to be reclaimed are determined from the local memory cache according to a preset cache eviction policy. If the candidate cache entry has the cache protection flag set, skip the recycling of the candidate cache entry, or migrate the candidate cache entry to the protected cache queue; If the candidate cache entry does not have the cache protection flag set, the candidate cache entry is reclaimed.

5. The directory-level data caching management method according to claim 1, characterized in that, The effective caching strategy includes caching modes, which include resident mode and priority mode. The step of skipping or delaying the reclamation of cache entries marked with the cache protection flag during the cache reclamation process of the local memory cache includes: If the local memory cache is insufficient, cache reclamation is performed according to the directory-aware hierarchical order. The directory-aware hierarchical order is as follows: first, cache entries that have not been marked with the cache protection flag are reclaimed; if the target release capacity has not been reached, cache entries with the cache mode set to the priority mode are reclaimed; if the target release capacity has not been reached and the degradation conditions are met, cache entries with the cache mode set to the resident mode are reclaimed.

6. The directory-level data caching management method according to claim 5, characterized in that, The effective caching strategy also includes the maximum cache capacity of the target directory. After setting the cache protection flag for the cache entry, it further includes: If the maximum cache capacity is non-zero and the protected cache capacity of the target directory exceeds the maximum cache capacity, the protected cache entry corresponding to the target directory is reclaimed according to at least one of access time, file priority, or object offset within the protected cache entry corresponding to the target directory.

7. The directory-level data caching management method according to claim 6, characterized in that, Also includes: If the degradation condition is met, the cache protection flag of the cache entry is removed, or the cache mode of the cache entry is adjusted from the resident mode to the priority mode; The downgrade condition is any one of the following: the maximum cache capacity is a non-zero value and the protected cache capacity corresponding to the target directory exceeds the maximum cache capacity; the available memory of the client is lower than the security threshold; a forced release instruction is received; the cache resident policy of the target directory is deleted or disabled; the data of the target file is modified, truncated, deleted, or its consistency fails; or the policy version of the local record is determined to be expired after reconnecting with the metadata server.

8. The directory-level data cache management method according to any one of claims 1 to 7, characterized in that, After setting the cache protection flag for the cache entry, the method further includes: If a truncation event, deletion event, renaming event, cache permission revocation event, or snapshot rollback event is detected for the target file, the corresponding cache entries are cleaned up or the cache protection flag of the corresponding cache entries is removed, based on the inode corresponding to the target file and the object scope.

9. The directory-level data caching management method according to any one of claims 1 to 7, characterized in that, Following the step of mapping the file offset range indicated by the access request to the object range of at least one data object, the method further includes: When the effective caching policy indicates that server-side cache protection is enabled, send an object read / write request carrying an object-level cache protection tag to the object storage service provider, or send an independent object protection request to the object storage service provider. The object-level cache protection tag or the object protection request is determined based on the object scope and the effective caching policy. It is used to instruct the object storage service to set a server-side cache protection tag for the server-side cache entries corresponding to the object scope, and to skip or delay the recycling of server-side cache entries with the server-side cache protection tag during the server-side cache recycling process.

10. A directory-level data cache management method, characterized in that, Metadata servers used in distributed file systems include: Receive a policy setting request for a target directory, and obtain the cache persistence policy configured for the target directory from the policy setting request; Based on the cache resident policy, determine the effective caching policy for the target files in the target directory; An effective caching policy for the target file is issued to the client, so that when the client receives an access request for the target file, it maps the file offset range indicated by the access request to the object range of at least one data object according to the file layout information of the target file. If the data corresponding to the object range is loaded into the cache entry of the client's local memory cache, or if the cache entry of the data corresponding to the object range is hit in the local memory cache, the client sets a cache protection flag for the cache entry according to the effective caching policy. The client is used to skip or postpone the reclamation of cache entries with the cache protection flag set during the cache reclamation process of the local memory cache.

11. The directory-level data caching management method according to claim 10, characterized in that, After the step of obtaining the cache resident policy configured for the target directory from the policy setting request, the method further includes: The validity of the policy fields in the cache persistence policy is verified. The policy fields include at least one of the following: server-side cache protection enabled status, client-side cache protection enabled status, reserved cache capacity, maximum cache capacity, cache priority, subdirectory inheritance identifier, cache mode, and cache eviction policy. The cache persistence policy that has passed the verification is persisted and a policy version corresponding to the cache persistence policy is generated.

12. The directory-level data caching management method according to claim 11, characterized in that, After the step of persistently storing the verified cache resident strategy, the method further includes: Maintain policy status statistics information corresponding to the cache resident policy. The policy status statistics information includes at least one of the following: client information holding the cache resident policy, protected cache capacity reported by each client, and protected cache capacity reported by the object storage service client.

13. The directory-level data caching management method according to claim 10, characterized in that, The step of determining an effective caching strategy for the target files in the target directory based on the caching persistence strategy includes: If the target file does not have a caching policy configured, search upwards through the parent directory of the target file for the directory with the most recently configured caching resident policy, and determine the caching resident policy configured in the found directory as the effective caching policy for the target file.

14. The directory-level data cache management method according to any one of claims 10 to 13, characterized in that, Also includes: If the cache persistence policy of the target directory is added, modified or deleted, update the policy version corresponding to the cache persistence policy and determine the list of clients that currently hold the inode or cache permissions of the files in the target directory. A policy update notification is sent to each client in the client list so that each client updates its effective caching policy for local records.

15. A client comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the directory-level data cache management method as described in any one of claims 1 to 9.

16. A metadata server, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the directory-level data cache management method as described in any one of claims 10 to 14.

17. A distributed file system, characterized in that, It includes the metadata server as described in claim 16, and at least one client as described in claim 15.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the directory-level data cache management method as described in any one of claims 1 to 14.

Images