Directory entry processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a directory entry processing method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the following steps: receiving an operation for creating a directory or newly adding a directory entry; the directory entry count is updated in response to the operation, the directory entry count is used for recording the total number of files and subdirectories in a target directory, and the target directory is a directory targeted by the operation; in a case where the value of the directory entry count satisfies a large directory condition, a continuous space including N blocks is allocated in the resource group, and a new directory entry generated based on the operation is stored in the continuous space, where N is greater than 1. According to the method provided by the invention, the continuity of metadata access is increased, and the traversal and statistical performance of the cluster file system during processing of massive file directories is improved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to a directory entry processing method, device, storage medium, and program product. Background Technology

[0002] With the development of distributed storage technology, cluster file systems (CFS) have become the core infrastructure for ultra-large-scale data storage scenarios. Users typically need to access large directories containing massive amounts of files and subdirectories frequently. When the number of files in a directory reaches tens of thousands or even hundreds of thousands, traditional metadata storage and access mechanisms struggle to meet the demands for high concurrency and low latency, becoming a key bottleneck restricting CFS performance.

[0003] In current implementations, CFS typically employs a distributed storage strategy when processing large directory metadata. Specifically, the system stores directory entries and their corresponding index nodes (inodes) discretely across multiple data blocks or different storage nodes to achieve distributed metadata management. During directory traversal or statistical operations, the system needs to traverse each directory entry and retrieve the inode information for each file by searching the index or mapping relationship.

[0004] However, the above implementation methods are difficult to meet the needs of efficient metadata access in scenarios with massive amounts of files. Summary of the Invention

[0005] This application provides a directory entry processing method, device, storage medium, and program product to solve the problem of efficient metadata access technology in scenarios with massive files.

[0006] Firstly, this application provides a directory entry processing method applied to a clustered file system, wherein the clustered file system includes resource groups, which are the basic management units for metadata and data storage. The method includes:

[0007] Received an operation for creating a directory or adding a directory entry;

[0008] In response to this operation, the directory entry count is updated. The directory entry count is used to record the total number of files and subdirectories under the target directory, which is the directory targeted by this operation.

[0009] If the value of the directory entry count satisfies the large directory condition, allocate a contiguous space consisting of N blocks in the resource group, and store the new directory entries generated based on this operation in the contiguous space, where N is greater than 1.

[0010] In this embodiment, when the system detects that the directory size has reached the large directory threshold through a dynamic counter, the storage mode for subsequently added metadata switches from the default random distributed mode to a centralized continuous mode. This provides the foundation for subsequent efficient large directory traversal operations, enabling metadata that originally required a large amount of random I / O readings to be quickly obtained with a small amount of sequential I / O, thereby fundamentally alleviating the I / O amplification problem.

[0011] In one possible implementation, the directory entry count value satisfies a large directory condition, including: the directory entry count value is greater than or equal to a preset value; the resource group contains a metadata area and a general data area, the metadata area is used to store the metadata of directory entries, and the general data area is used to store the data of general files; a contiguous space comprising N blocks is allocated in the resource group, including:

[0012] Allocate a contiguous space comprising N blocks in the metadata area of ​​the resource group.

[0013] In this implementation, the metadata of large directories is centrally and continuously stored in a dedicated physical area through the collaborative design of storage structure, identification logic, and allocation mechanism. This changes the random and scattered layout of metadata in traditional solutions, transforming the large number of random disk I / O operations required for subsequent directory traversal or space statistics of massive files into a small number of efficient sequential I / O operations, thereby improving the system's metadata access performance in large directory scenarios.

[0014] In one possible implementation, the method further includes:

[0015] Received a directory traversal instruction for the target directory;

[0016] Based on directory traversal instructions, the metadata of directory entries is obtained. The metadata of directory entries includes the name of the directory entry and the address of the index block.

[0017] If the contiguous space containing the address of the index block is not pre-read, asynchronously pre-read the metadata of the file information in the contiguous space containing the address of the index block, and mark the contiguous space containing the address of the index block as pre-read.

[0018] This implementation combines a bitmap mechanism with asynchronous prefetching to achieve a one-time loading of large-scale contiguous metadata regions. When traversing a large directory containing a massive number of files, all access requests for inode information are satisfied by the cache, thereby transforming disk I / O from a large number of random single-point reads to a small number of sequential prefetches, improving the response speed of directory traversal operations and the overall system efficiency.

[0019] In one possible implementation, during the asynchronous pre-reading of metadata of file information in the contiguous space where the address of the index block is located, a distributed lock for all metadata in the contiguous space where the address of the index block is located is requested in batches; and then, the distributed lock is asynchronously waited for to be granted.

[0020] This implementation optimizes batch requests and asynchronous waiting, reducing the inherent distributed lock negotiation overhead—proportional to the number of files—in massive metadata access scenarios to a near-constant overhead related to the number of pre-read intervals. This addresses the amplification effect of network latency and lock contention on metadata access performance in clustered environments, enabling efficient implementation of pre-read strategies based on contiguous storage in distributed environments, thereby improving the performance of large directory traversals.

[0021] In one possible implementation, the cluster file system includes: a virtual file system layer, a cluster file system layer, and a distributed lock layer;

[0022] Receive directory traversal instructions for the target directory, including: receiving directory traversal instructions at the virtual file system layer;

[0023] Based on directory traversal commands, retrieve the metadata of directory entries, including:

[0024] The virtual file system layer calls the interface of the cluster file system layer, enabling the cluster file system layer to obtain directory traversal instructions.

[0025] The cluster file system layer requests a distributed lock for the target directory from the distributed lock layer.

[0026] Retrieve the source data for the target directory from the cache or disk, and release the distributed lock on the target directory.

[0027] This implementation clearly defines a three-layer collaborative architecture consisting of a virtual file system layer, a clustered file system layer, and a distributed lock layer. It also restricts the release of directory locks to the point where basic directory entry information is obtained, achieving the following technical effects: Firstly, it standardizes and decouples the system processing flow, improving modularity and maintainability. Secondly, it shortens the holding time of directory locks, effectively alleviating lock contention under high concurrency, thereby improving system throughput and concurrency performance, and providing a reliable data foundation for subsequent pre-read optimization.

[0028] In one possible implementation, the clustered file system also includes: a caching layer, and metadata about file information in the contiguous space containing the addresses of asynchronously read index blocks, including:

[0029] Determine the type of the blocks within the contiguous space where the address of the pre-read index block is located, and then filter out the index blocks;

[0030] Request distributed locks for index blocks in batches and wait asynchronously for the distributed locks to be granted;

[0031] Once the distributed lock is acquired, read the metadata of file information in the index blocks in batches;

[0032] The file information metadata is cached in the cache layer, and the distributed lock is released.

[0033] This implementation introduces a caching layer and refined asynchronous prefetching tasks, transforming the access mode from passive disk reading to active memory provisioning. After obtaining basic directory entry information, the system asynchronously and in batches prefetches metadata from contiguous storage space and caches it. This allows subsequent requests for detailed file information to be retrieved directly from memory, transforming a large number of random disk I / O operations into efficient memory access. This reduces traversal latency, improves user experience, and, through batch lock acquisition and asynchronous processing mechanisms, reduces lock negotiation overhead in a distributed environment, improving the overall throughput and concurrency performance of the system in a cluster environment.

[0034] In one possible implementation, the method further includes at least one of the following:

[0035] The preset values ​​used to determine whether the large directory condition is met are dynamically adjusted based on the load status of the cluster file system.

[0036] The preset value is adjusted according to the growth rate of the directory entries in the target directory. When the growth rate of the directory entries exceeds the preset rate threshold, the preset value is reduced.

[0037] This implementation introduces an adaptive feedback mechanism based on system load and directory growth rate, transforming the identification and optimization strategy for large directories from static configuration to dynamic decision-making. The system can flexibly adjust the judgment threshold based on real-time load, proactively lowering the threshold under high load to prevent performance bottlenecks and raising the threshold under low load to conserve storage resources. Simultaneously, by monitoring the instantaneous growth rate of directories, the system quickly lowers the threshold for directories with high-speed writes, thereby enabling continuous storage optimization earlier and accurately responding to business peaks. This dynamic tuning mechanism, while ensuring storage efficiency, enhances the system's resource adaptability under different loads and business scenarios.

[0038] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0039] The memory stores the instructions that the computer executes;

[0040] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0041] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0042] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 A flowchart illustrating a catalog item processing method provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating the process of determining the large directory and allocating inodes sequentially, provided for embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the resource group structure and large directory inode allocation provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating a scenario representing memory bitmap markers provided in an embodiment of this application;

[0048] Figure 5 This is a flowchart illustrating the execution of large directory traversal and pre-read tasks in a cluster file system provided in this embodiment of the application.

[0049] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0051] It should be noted that the catalog entry processing method, device, storage medium, and program product provided in this application can be used in the distributed field, or in any field other than distributed. This application does not limit the application field of the catalog entry processing method, device, storage medium, and program product.

[0052] Specific application scenarios of this application include, for example, performance bottlenecks caused by accessing massive amounts of file metadata in CFS, especially for large-scale, high-concurrency distributed data storage environments. Typical application scenarios include, but are not limited to:

[0053] Enterprise-level data analytics platforms: In data lakes and data warehouses, millions of data files typically accumulate under a single log partition or fact table directory. Frequent data exploration, job scheduling, and storage space auditing require high-frequency listing and space statistics operations on large directories.

[0054] In high-performance computing and scientific research: In scenarios such as bioinformatics, meteorological simulation, and physical computing, a single computational task may generate massive amounts of intermediate or result files, such as sequence files and slice data, all stored centrally in the same working directory. Researchers need to quickly list or analyze these files for subsequent analysis or resource management.

[0055] Cloud storage and backup archiving systems: Cloud-native applications and backup archiving systems typically organize massive amounts of objects in a flat or hierarchical structure. When users browse directories or query capacity through clients or management consoles, the background triggers a centralized traversal request for the metadata of large directories.

[0056] To address the metadata access needs in the aforementioned large-scale data storage scenarios, CFS currently commonly employs the following technical solutions:

[0057] Distributed metadata storage architecture: Existing systems typically manage file metadata using a distributed storage approach. Specifically, file inode information is discretely stored across multiple data blocks in the file system, with no inherent physical association between each inode and its corresponding file data block. As the directory size increases, the inodes of each file are distributed across different areas of the storage medium according to a general allocation algorithm. The system maintains the association between the file's logical structure and physical storage through a mapping relationship between directory entries and inode numbers.

[0058] Locality-based metadata prefetching mechanism: Some systems implement prefetching strategies to accelerate metadata access. This strategy is mainly designed based on the access patterns of file data blocks. For example, when the system detects continuous reading of file data, it will attempt to prefetch the metadata corresponding to adjacent data blocks. The prefetching window size is usually set to a fixed value, and the triggering condition is based on the traditional spatial locality access assumption.

[0059] Unified storage resource allocation strategy: In terms of storage resource management, the existing system uses the same metadata space allocation strategy for all directories. The system does not distinguish between ordinary directories and large directories; all inodes are allocated on demand from a unified storage pool. Under this design, the metadata of large directories is stored in the same physical area along with the metadata of other directories.

[0060] A distributed coordination mechanism based on fine-grained locks: In a clustered deployment environment, metadata access ensures consistency through a distributed lock protocol. The system initiates a separate lock negotiation process for each metadata operation (including read, modify, and read-ahead operations) to ensure data correctness when accessed by multiple nodes. This mechanism requires multiple network communications between the client and the metadata server to acquire and release locks.

[0061] However, the solution described above has the following technical problems:

[0062] The problem of IO amplification caused by the scattered storage of inodes in large directories: In the existing technology, the inode information of large directories is stored randomly. When traversing the directory, frequent discrete IO requests are required, which leads to a surge in the number of IO operations and prolongs the time spent traversing the directory.

[0063] Frequent disk reads caused by low metadata cache hit rate: Existing technologies lack efficient pre-read strategies for large directories. Access to missed inodes requires triggering disk read operations, and the low cache hit rate further exacerbates latency.

[0064] The high overhead of distributed lock negotiation in a cluster environment: In CFS, access to missed metadata needs to be coordinated through distributed locks. However, existing technologies do not optimize lock requests in pre-read tasks in batches, which increases the lock negotiation time and amplifies the performance loss.

[0065] The directory entry processing method provided in this application aims to solve the aforementioned technical problems of the prior art. By dynamically identifying large directories and centrally storing their metadata in reserved contiguous physical space, the storage layout of metadata is fundamentally changed. Based on this physical optimization, an asynchronous pre-read mechanism is used to pre-load centralized inode information into the cache, and a bitmap mechanism is used to avoid repeated pre-reads. At the same time, in a distributed environment, the metadata access locks required for pre-read operations are batch-applied and asynchronously processed, thereby achieving collaborative optimization at the storage layer, cache layer, and coordination layer. Ultimately, the directory traversal operation of massive files is transformed from a large number of discrete random I / Os to a small number of continuous I / Os and memory accesses, thereby reducing disk I / O, network latency, and lock negotiation overhead.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] First, combine Figure 1 The flowchart of the catalog item processing method provided in the embodiments of this application will be described. Figure 1 This application provides a flowchart illustrating a directory entry processing method. The method is applied to CFS, and the corresponding basic storage management unit is a resource group. The specific steps of the method include:

[0068] S101, Received an operation for creating a directory or adding a directory entry.

[0069] The CFS metadata server receives requests from clients to create directories or add directory entries. CFS storage space is divided into multiple independent resource groups according to a preset size, serving as the basic management unit for metadata and data storage.

[0070] S102, Update the directory entry count in response to the operation.

[0071] In response to this operation, the system updates the directory entry count of the target directory. This count, maintained in the directory's inode metadata, is used to count the total number of files and subdirectories directly contained in the target directory. The system then determines whether the count value meets the large directory condition, which in this embodiment is specifically manifested as: the count value is greater than a preset threshold (e.g., 128). When the count value exceeds this threshold for the first time, the system automatically marks the directory as a "large directory type".

[0072] S103. If the value of the directory entry count satisfies the large directory condition, allocate a contiguous space comprising N blocks in the resource group, and store the new directory entries generated based on the operation in the contiguous space.

[0073] If the directory entry count does not meet the large directory condition, the system will process new directory entries according to the normal strategy.

[0074] If the directory entry count meets the large directory condition, the system executes an optimization strategy: allocating contiguous space in the metadata reservation area of ​​the current resource group. This reservation area has a fixed total capacity (e.g., 8192 disk blocks) and is dedicated to storing the inodes of the large directory. Allocation is done in units of 128 contiguous disk blocks. The system divides this reservation area into N (N≥1, typically starting with 128) contiguous blocks at once, storing the new directory entries and their inodes generated in this operation within this contiguous space. If the currently allocated 128-block area is full, the system will continue to allocate the next 128 contiguous blocks within the reservation area, ensuring that all inodes under the same large directory are physically stored contiguously.

[0075] The directory entry processing method provided in this embodiment optimizes the storage layout of metadata for large directories. When the system detects that the directory size has reached the large directory threshold through a dynamic counter, the storage mode for subsequently added metadata switches from the default random and distributed mode to a centralized and continuous mode. This provides a foundation for efficient large directory traversal operations, enabling metadata that originally required a large amount of random I / O readings to be quickly obtained with a small amount of sequential I / O, thereby fundamentally alleviating the I / O amplification problem.

[0076] To facilitate understanding of the directory item processing flow in the above embodiments, the following will combine... Figure 2 The complete execution logic of the above method will be introduced. Figure 2 This is a schematic diagram illustrating the process of determining the large directory and allocating inodes sequentially, as provided in an embodiment of this application.

[0077] like Figure 2 As shown, the process begins with the client initiating a request to create a directory or add a directory entry. After receiving the request, the system first performs an update directory entry counter step, which increments the directory entry counter of the target directory by 1. This counter is used to accurately count the total number of files and subdirectories directly contained in the directory.

[0078] The process then proceeds to the core decision node: determining whether the directory item count is greater than 128. Here, 128 is a pre-set threshold, corresponding to the specific numerical standard for determining whether the large directory condition is met in the above embodiment.

[0079] Based on the judgment result, the process generates the following two branches:

[0080] Branch 1 (No), Directory Item Count ≤ 128:

[0081] This branch indicates that the target directory is a regular directory. In this case, the system will process the new directory entry according to the usual strategy, storing the inode in the regular data area of ​​the corresponding resource group, using a traditional, discrete storage management method.

[0082] Branch 2 (Yes), Directory Item Count > 128:

[0083] This branch indicates that the target directory has reached the large directory standard, and the system will perform the following optimization process:

[0084] Mark as a large directory: Mark the metadata attributes of the target directory as "large directory type".

[0085] Allocate 128 contiguous blocks of space: Allocate 128 contiguous disk blocks as storage units at once from the dedicated inode reservation area (i.e., metadata reservation area) of the resource group to which the target directory belongs.

[0086] Store inodes in contiguous space: The new directory entry generated in this operation, along with its complete inode metadata, is stored in the contiguous space allocated above. If this 128 contiguous space blocks are full, the system will automatically allocate the next contiguous 128 blocks from the same reserved area for subsequent storage, thus ensuring that all inodes under the same large directory are physically stored contiguously.

[0087] Therefore, by dynamically monitoring and comparing with preset thresholds, the system can automatically identify large directories, thereby triggering the switching of metadata storage strategies, changing the physical layout from a random discrete distribution to a centralized continuous distribution, and providing a physical foundation for subsequent efficient directory traversal and access.

[0088] Optionally, this embodiment further illustrates a preferred implementation of directory item processing, which will be discussed below in conjunction with... Figure 3 The implementation of the file system storage unit and decision-making logic is introduced. Figure 3 This is a schematic diagram of the resource group structure and large directory inode allocation provided in an embodiment of this application.

[0089] like Figure 3 As shown, this implementation employs a fine-grained hierarchical design for the storage resources of the cluster file system to adapt to the efficient metadata access requirements in large directory scenarios. The specific implementation includes the following three layers:

[0090] Structured organization of storage resources:

[0091] At the CFS level, the physical storage pool is logically divided into multiple independent resource groups (e.g., resource group 0, resource group 1, etc.), with each resource group serving as its own basic management unit. Figure 3 This diagram details the internal architecture of a resource group, as shown in resource group 0 in the figure, which comprises the following four key components:

[0092] Resource group metadata: Used to record the description and management information of the resource group itself.

[0093] Bitmap: Tracks the free and occupied status of each storage block within the resource group in bitmap form.

[0094] Inode Reserved Area (Dedicated Metadata Area): This area is a contiguous storage area with a fixed total capacity (e.g., 8192 disk blocks). It is used only to store the inodes corresponding to directory entries identified as "large directories". This design physically achieves complete isolation between large directory metadata and ordinary directory inodes and file data.

[0095] General data area: Used to store the actual file content data.

[0096] Dynamic identification mechanism for large directories:

[0097] To trigger the aforementioned storage policy, the system maintains a directory entry counter for each directory, continuously counting the total number of direct sub-entries (files and subdirectories) under each directory, and sets a configurable preset threshold. When the system detects that the counter value of a directory reaches or exceeds this threshold, it dynamically identifies and marks the directory as a "large directory".

[0098] Continuous allocation strategy based on dedicated areas:

[0099] For large directories that have already been identified, the inode storage for subsequently added directory entries will point to... Figure 3 The reserved area structure is shown. Internally, this reserved area structure uses 128 consecutive disk blocks as a basic allocation unit. For example... Figure 3 As shown, when space needs to be allocated for a large directory, the system does not allocate a single block, but rather divides one or more contiguous basic units from the reserved area structure at once (i.e., allocates N blocks, where N≥1 and is usually a multiple of 128). This strategy of allocating in groups ensures that all inodes under a large directory can be stored as centrally and contiguously as possible.

[0100] Through the collaborative design of the aforementioned storage structure, identification logic, and allocation mechanism, the metadata of large directories is centrally and continuously stored in a dedicated physical area. This changes the traditional layout of randomly distributed metadata, transforming the large number of random disk I / O operations required for subsequent directory traversal or space statistics of massive files into a small number of efficient sequential I / O operations, thereby improving the system's metadata access performance in large directory scenarios.

[0101] Optionally, based on the resource group structure optimization, this embodiment further demonstrates an implementation method to improve the performance of large directory traversal through asynchronous prefetching and memory bitmap state marking. Its core lies in utilizing a bitmap mechanism to ensure that an efficient prefetch is performed only once for each contiguous inode storage region.

[0102] In the first stage of traversing the large directory, once the system obtains the inode block address of the directory entry, it immediately triggers a pre-read process. At this time, the system will determine the 128 contiguous blocks to which it belongs based on this address, which is the smallest unit of inode allocation described in the above embodiment. The following will combine... Figure 4 The implementation of the bitmap mechanism will be introduced. Figure 4 This is a schematic diagram of a scenario representing a memory bitmap marker, provided in an embodiment of this application.

[0103] like Figure 4 As shown, the system maintains a prefetch status bitmap in memory. Each bit in this bitmap uniquely corresponds to a contiguous storage region of 128 blocks, forming a direct mapping relationship. The diagram clearly indicates the two states of each bit: "0" represents that the region has not been prefetched, and "1" represents that the region has been prefetched. The specific workflow is as follows:

[0104] When it is necessary to read the range where a certain inode is located, the system first queries the status of the corresponding bit in the bitmap.

[0105] If the bit is "0", it indicates that the inode metadata in this contiguous region has not yet been loaded into the cache. The system then initiates an asynchronous prefetch task to read all the inode information in the 128 contiguous regions into the cache in batches. After the prefetch task is completed, the corresponding bit in the bitmap is updated to "1" to mark that the interval has been prefetched.

[0106] If this bit is "1", it indicates that the metadata for that range already exists in the cache. The system will skip the prefetch operation and directly utilize the cached data, thereby avoiding repeated I / O accesses to the same disk area.

[0107] Furthermore, to ensure the consistency of cached data, when any inode metadata within a contiguous region of 128 blocks is updated, the system proactively resets the corresponding bit in the bitmap to "0". This mechanism allows subsequent accesses to detect data changes and trigger prefetching again to obtain the latest information.

[0108] By combining the bitmap mechanism described above with asynchronous prefetching, a one-time loading of large-scale contiguous metadata regions is achieved. When traversing a large directory containing a massive number of files, all access requests for inode information will be satisfied by the cache, thereby transforming disk I / O from a large number of random single-point reads to a small number of sequential prefetches, improving the response speed of directory traversal operations and the overall system efficiency.

[0109] Optionally, this embodiment further illustrates a lock negotiation optimization scheme for distributed environments based on the asynchronous pre-read mechanism, which is used to solve the coordination overhead problem of metadata pre-read in a cluster environment.

[0110] When a read-ahead task is triggered, requiring the reading of metadata from a contiguous storage unit (e.g., a read-ahead interval of 128 blocks), the system executes the following process:

[0111] Batch lock requests: The prefetch task first requests distributed locks for all metadata (inodes) within the target contiguous storage unit in batches. For example, when prefetching a range of 128 blocks, the system generates a unified lock request covering all inodes within that range and submits it to the distributed lock service all at once, thereby merging the traditionally discrete and multiple lock negotiations into a single coarse-grained coordination operation.

[0112] Asynchronous lock waiting and process decoupling: After the lock request is submitted, the system adopts an asynchronous waiting strategy to decouple the process and reduce latency.

[0113] Specifically, the main process (such as a directory traversal request) is not blocked; it can continue or return, only needing to synchronously wait for the individual metadata locks currently required. For example, the inode lock of the first file being accessed.

[0114] The prefetch task, acting as a background thread, asynchronously waits for the granting of all remaining inode locks in the aforementioned batch request. This prevents the main process from being suspended for an extended period due to waiting for a large number of locks.

[0115] Data Reading and Lock Release: Once the prefetch task asynchronously acquires the batch lock, it performs batch I / O operations, reading all metadata within the contiguous unit into the cache. After completion, these lock resources are released in batches.

[0116] Lock state and cache consistency linkage: To ensure data consistency, when any metadata in the contiguous storage unit is updated, the system will not only reset the pre-read bitmap state as described in the above embodiments, but also trigger the corresponding metadata lock state to reset or invalidate, so as to ensure that subsequent pre-reads can re-acquire the latest data.

[0117] By optimizing batch requests and asynchronous waiting, the inherent distributed lock negotiation overhead in massive metadata access scenarios, which is proportional to the number of files, is compressed into a near-constant overhead related to the number of pre-read intervals. This solves the problem of network latency and lock contention amplifying the impact on metadata access performance in clustered environments, enabling efficient implementation of pre-read strategies based on contiguous storage in distributed environments, thereby improving the performance of large directory traversal.

[0118] Optionally, this embodiment also illustrates the core interaction process for handling directory traversal requests in a three-layer architecture consisting of a virtual file system (VFS) layer, a clustered file system layer, and a distributed lock layer. The following will combine... Figure 5This section introduces the interaction process. Figure 5 This is a flowchart illustrating the execution of large directory traversal and pre-read tasks in a cluster file system provided in this embodiment of the application.

[0119] like Figure 5 As shown in "Directory Entry Traversal Phase One," when a directory traversal command is initiated by user space, it is first received at the virtual file system layer. The virtual file system layer then calls the interface of the cluster file system layer to pass the directory traversal command to the cluster file system layer.

[0120] Upon receiving a directory traversal instruction, the cluster file system layer requests a distributed lock for the target directory from the distributed lock layer to ensure data access consistency. After successfully acquiring the distributed lock for the directory, the cluster file system layer performs the following operations:

[0121] The target directory's metadata is retrieved from the cache or disk to parse it, thereby obtaining the names of all directory entries and their corresponding inode block addresses. Once the target directory's metadata is retrieved, the cluster file system layer releases the distributed lock on the target directory and returns a preliminary list containing the directory entry names.

[0122] At this point, the process of obtaining basic metadata of directory entries (including directory entry name and inode block address) based on directory traversal instructions is complete.

[0123] By defining a three-layer collaborative architecture consisting of a virtual file system layer, a clustered file system layer, and a distributed lock layer, and limiting the release of directory locks to after acquiring basic directory entry information, the following technical effects were achieved: On the one hand, the system processing flow was standardized and decoupled, improving modularity and maintainability; on the other hand, the holding time of directory locks was shortened, effectively alleviating lock contention under high concurrency, thereby improving system throughput and concurrency performance, and providing a reliable data foundation for subsequent pre-read optimization.

[0124] Optionally, this embodiment further introduces a caching layer based on the three-layer architecture of the above embodiments, and according to... Figure 5 The specific execution process of the asynchronous read-ahead task in the stage of obtaining file details is shown in detail.

[0125] like Figure 5 As shown in the "Asynchronous Execution of Prefetch Tasks" section, after obtaining the basic information of directory entries, the cluster file system layer triggers an asynchronous prefetch task for operations that require reading the contents of specific directory entry inode blocks. This task is executed independently of the main process, and the specific steps are as follows:

[0126] Judgment and Filtering: The asynchronous task layer determines the block type within the pre-read interval and filters out the inode blocks. The pre-read interval refers to the contiguous storage space where the currently accessed inode block address is located.

[0127] Batch lock processing: The asynchronous task layer requests distributed locks for all selected inode blocks from the distributed lock layer in batches, and enters an asynchronous waiting state for lock granting. This process does not block the main request flow.

[0128] Batch Read and Caching: After successfully acquiring the batch lock, the asynchronous task layer reads the file information metadata from the inode blocks in batches and caches the read file information metadata in the cache layer. After caching is complete, the distributed locks on these inode blocks are released in batches.

[0129] like Figure 5 As shown in "Directory Entry Traversal Phase Two", when the user process or the system needs to obtain detailed information such as file size and modification time, the system directly obtains the pre-stored complete inode metadata from the cache layer without having to access the disk again.

[0130] By introducing a caching layer and refined asynchronous prefetching tasks, a shift from passive disk reads to proactive memory provisioning is achieved. After obtaining basic directory entry information, the system asynchronously and in batches prefetches metadata from contiguous storage space and caches it. This allows subsequent requests for detailed file information to be retrieved directly from memory, transforming a large number of random disk I / O operations into efficient memory access. This reduces traversal latency, improves user experience, and, through batch lock acquisition and asynchronous processing mechanisms, reduces lock negotiation overhead in a distributed environment, thereby improving the overall throughput and concurrency performance of the system in a cluster environment.

[0131] Optionally, this embodiment also provides an adaptive threshold tuning mechanism, which enables the identification strategy of large directories to flexibly respond to changes in system operating status and business load, thereby achieving a dynamic balance between storage efficiency and access performance.

[0132] Specifically, the system can implement at least one of the following tuning strategies:

[0133] Strategy 1: Dynamic adjustment based on system load status

[0134] The system monitors the load status of the cluster file system in real time, including key indicators such as CPU utilization, disk I / O throughput, and network bandwidth utilization. Based on the monitoring results, the system dynamically adjusts the preset value used to determine large directories; for example, the preset value is 128.

[0135] For example, when the system is under high load (such as CPU utilization consistently above 80%), it indicates that system resources are strained and the system is more sensitive to performance bottlenecks. In this case, the system can appropriately reduce the preset value (e.g., from 128 to 100) so that more medium-sized directories can be identified as "large directories" in advance and enable continuous metadata storage optimization strategies, thereby more effectively preventing potential performance degradation.

[0136] Conversely, under low load, the system can appropriately increase the preset value to avoid allocating dedicated contiguous resources to excessively small directories, thereby making better use of storage space.

[0137] Strategy 2: Dynamic adjustment based on directory growth rate

[0138] The system continuously monitors and calculates the growth rate of directory entries in the target directory, which is the number of new files or subdirectories added per unit of time. Based on this growth rate, it dynamically adjusts the preset value used to determine whether the directory meets the criteria for a large directory. When the growth rate of directory entries is detected to exceed the preset threshold, the preset value used for that directory is proactively reduced.

[0139] For example, the preset growth rate threshold can be set to 10 new files per second. If the system detects that the instantaneous growth rate of a directory reaches 50 files per second, it is determined to be a high-growth directory. The system then lowers the preset value for large directories from the standard value of 128. This allows the directory to be identified as a large directory more quickly, thereby enabling continuous metadata storage optimization strategies for the directory earlier to cope with the possible massive file size and avoid performance bottlenecks caused by recognition delays.

[0140] By introducing an adaptive feedback mechanism based on system load and directory growth rate, the identification and optimization strategy for large directories has been transformed from static configuration to dynamic decision-making. The system can flexibly adjust the judgment threshold based on real-time load, proactively lowering the threshold under high load to prevent performance bottlenecks and raising the threshold under low load to conserve storage resources. Simultaneously, by monitoring the instantaneous growth rate of directories, the system quickly lowers the threshold for directories with high-speed writes, thereby enabling continuous storage optimization earlier and accurately responding to business peaks. This dynamic tuning mechanism, while ensuring storage efficiency, enhances the system's resource adaptability under different loads and business scenarios.

[0141] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0142] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0144] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0145] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0149] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0150] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0151] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0152] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0153] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0154] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0155] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for processing directory entries, characterized in that, Applied to a clustered file system, wherein the clustered file system includes resource groups, and the resource groups are the basic management units for metadata and data storage, the method includes: Received an operation for creating a directory or adding a directory entry; In response to the operation, the directory entry count is updated. The directory entry count is used to record the total number of files and subdirectories under the target directory, which is the directory to which the operation is applied. If the value of the directory entry count satisfies the large directory condition, a contiguous space comprising N blocks is allocated in the resource group, and new directory entries generated based on the operation are stored in the contiguous space, where N is greater than 1.

2. The method according to claim 1, characterized in that, The value of the directory entry count satisfies the large directory condition, including: the value of the directory entry count is greater than or equal to a preset value; the resource group includes a metadata area and a normal data area, the metadata area is used to store the metadata of directory entries, and the normal data area is used to store the data of ordinary files; the allocation of contiguous space comprising N blocks in the resource group includes: Allocate a contiguous space comprising N blocks in the metadata area of ​​the resource group.

3. The method according to claim 2, characterized in that, The method further includes: Received a directory traversal instruction for the target directory; Based on the directory traversal instructions, the metadata of the directory entries is obtained, including the name of the directory entry and the address of the index block; If the contiguous space where the address of the index block is located is indicated as not pre-read, the metadata of the file information in the contiguous space where the address of the index block is located is asynchronously pre-read, and the contiguous space where the address of the index block is located is marked as pre-read.

4. The method according to claim 3, characterized in that, During the asynchronous pre-reading of metadata of file information in the contiguous space where the address of the index block is located, a distributed lock for all metadata in the contiguous space where the address of the index block is located is requested in batches. And, asynchronously wait for the distributed lock to be granted.

5. The method according to claim 3 or 4, characterized in that, The cluster file system includes: a virtual file system layer, a cluster file system layer, and a distributed lock layer; Receiving a directory traversal instruction for the target directory includes: receiving the directory traversal instruction at the virtual file system layer; The step of obtaining the metadata of directory entries based on the directory traversal instructions includes: The virtual file system layer calls the interface of the cluster file system layer, enabling the cluster file system layer to obtain the directory traversal instruction. The cluster file system layer requests a distributed lock for the target directory from the distributed lock layer. Retrieve the source data of the target directory from the cache or disk, and release the distributed lock of the target directory.

6. The method according to claim 5, characterized in that, The cluster file system further includes: a caching layer, wherein the metadata of file information in the contiguous space where the address of the index block is located during asynchronous pre-reading includes: Determine the type of blocks within the contiguous space containing the address of the index block to be read, and filter out the index blocks; Request distributed locks for the index blocks in batches and wait asynchronously for the distributed locks to be granted; If the distributed lock is acquired, the metadata of the file information in the index block is read in batches; The metadata of the file information is cached in the cache layer, and the distributed lock is released.

7. The method according to any one of claims 1-4, characterized in that, The method further includes at least one of the following: Based on the load status of the cluster file system, dynamically adjust the preset values ​​used to determine whether the large directory condition is met; The preset value is adjusted according to the growth rate of the directory entries in the target directory, wherein the preset value is reduced when the growth rate of the directory entries exceeds the preset rate threshold.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images