A file request processing method and an electronic device

Abstract

The application discloses a file request processing method and an electronic device, and relates to the technical field of storage systems, and simplifies lock management in a cluster file system by routing file requests to corresponding file home nodes. When a file request is received, an index node identifier is parsed according to a file path, so that an allocation group in which the file is located is determined. Then, the allocation group is mapped to a corresponding file home node by using a global view, and the request is forwarded to the node for processing. Since all operations on the same file are routed to the same home node, the node can use a simple local lock mechanism to ensure data consistency without complex distributed lock coordination, so that lock management logic is simplified, network communication overhead is reduced, and the overall performance and reliability of the system are improved.

Description

Technical Field

[0001] This application relates to the field of storage system technology, and in particular to a method for processing file requests and an electronic device. Background Technology

[0002] With the rapid development of information technology, data volume is exploding, and large-scale data storage and processing scenarios are placing higher demands on the performance, reliability, and scalability of storage systems. Clustered file systems, as a core component of distributed storage, can integrate the storage resources of multiple nodes, providing users with a unified file access interface and effectively meeting the needs of large-scale data storage. Therefore, they are widely used in fields such as cloud computing, big data analytics, and high-performance computing.

[0003] Currently, mainstream cluster file systems, such as those based on shared disk architecture, achieve data consistency between nodes through distributed locks, and also involve mechanisms such as glock state machines and range locks, resulting in complex lock management logic. Summary of the Invention

[0004] This application provides a method for processing file requests and an electronic device to at least solve the problem of complex lock management logic in related technologies.

[0005] This application provides a method for processing a file request, comprising: receiving a file request, the file request being used to request operation on a target file, the file request containing a file path of the target file; responding to the file request, resolving the file path to obtain an inode identifier corresponding to the target file, the inode identifier being used to indicate the offset position of the target file on a target allocation group in a target logical volume; determining a first file owner node corresponding to the target allocation group based on a global view and the identifier of the target allocation group; wherein, the global view contains the correspondence between each allocation group and multiple file owner nodes; and forwarding the file request to the first file owner node for processing the file request by the first file owner node.

[0006] This application also provides another method for processing file requests, applied to a file ownership node. The method includes: receiving a file request, the file request being used to request operation on a target file, the file request containing the file path of the target file; responding to the file request, resolving the file path to obtain the inode identifier corresponding to the target file, the inode identifier being used to indicate the offset position of the target file on the target allocation group in the target logical volume; accessing the target file in local storage according to the inode identifier, and performing the corresponding operation of the file request on the target file; and returning the execution result to the sender of the file request.

[0007] This application also provides a document request processing apparatus, comprising:

[0008] The receiving module is used to receive file requests. The file request requests the target file to be operated on and contains the file path of the target file.

[0009] The parsing module is used to respond to file requests by parsing the file path to obtain the inode identifier corresponding to the target file. The inode identifier is used to indicate the offset position of the target file on the target allocation group in the target logical volume.

[0010] The determination module is used to determine the first file ownership node corresponding to the target allocation group based on the global view and the identifier of the target allocation group; wherein, the global view contains the correspondence between each allocation group and multiple file ownership nodes;

[0011] The sending module is used to forward file requests to the first file-owning node so that the first file-owning node can process the file requests.

[0012] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described file request processing method.

[0013] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described file request processing method.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described file request processing method.

[0015] This application simplifies lock management in a clustered file system by routing file requests to the corresponding file-owning node. Upon receiving a file request, the inode identifier is first resolved from the file path to determine the allocation group to which the file belongs. Then, a global view is used to map this allocation group to the corresponding file-owning node, and the request is forwarded to that node for processing. Since all operations on the same file are routed to the same owner node, this node can use a simple local locking mechanism to ensure data consistency without complex distributed lock coordination. This design avoids cross-node lock contention and state synchronization, eliminates reliance on complex mechanisms such as glock state machines and range locks, simplifies lock management logic, reduces network communication overhead, and improves the overall performance and reliability of the system. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for processing a file request provided in an embodiment of this application;

[0018] Figure 2 A flowchart illustrating another method for processing file requests provided in an embodiment of this application;

[0019] Figure 3 A schematic diagram of a document request processing apparatus provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To more clearly illustrate the embodiments of this application, the technical terms used in the embodiments will be briefly introduced below:

[0024] A cluster file system is a file system that can be mounted and accessed by multiple servers simultaneously. Multiple machines can perform read and write operations on the same set of files, and the user experience is similar to that of a local file system for users and applications.

[0025] Global File System 2 (GFS2) is an open-source clustered file system designed for shared storage cluster environments, where all nodes directly access the shared underlying storage. It allows multiple nodes to concurrently access the same block device within a cluster and provides a consistent file system view across all nodes.

[0026] Oracle Cluster File System 2 (OCFS2) is an open-source cluster file system designed for shared storage cluster environments. It allows multiple nodes to share the same disk storage and provides a consistent file system view across all nodes.

[0027] A Distributed Lock Manager (DLM) is a software component in a distributed system that provides synchronization services, enabling multiple nodes to coordinate access to shared resources. It manages locks on resources such as files, database records, or other shared objects to ensure mutual exclusion and maintain data consistency across the entire cluster.

[0028] Lustre is a parallel distributed file system commonly used in large-scale cluster computing. It is designed to provide high performance, scalability, and reliability for applications requiring high I / O throughput.

[0029] A metadata server (MDS) is a component of a distributed file system used to store and manage metadata, such as filenames, directory structures, permission information, and file-to-data-block mappings. Clients retrieve metadata through the MDS, while actual file I / O operations interact directly with the data server.

[0030] In storage systems, logical volumes are typically designated by Logical Unit Numbers (LUNs). A LUN is a unified, manageable block-level storage entity formed by abstracting and aggregating the underlying physical storage resources. The LUN serves a dual role: a unique identifier for the logical volume (assigned by the storage system) and an access point, enabling host servers to recognize it as a local disk and perform mapping and interaction through the LUN identifier. The host does not need to be aware of the complex structure of the underlying physical storage; it can access the storage resources corresponding to the logical volume simply through block-level protocols such as Small Computer System Interface (SCSI), Internet Small Computer System Interface (iSCSI), and Fibre Channel, using the LUN as a means.

[0031] An Allocation Group (AG) contains file metadata and data, and can be understood as a small file system. It is a logical subdivision of the total storage capacity of the file system, designed to achieve parallel management of storage resources and concurrent I / O operations. Each AG, as an independent and self-contained unit, manages its own metadata (such as inodes, directory structure, and free space mapping tables) and physical storage blocks. By dividing the file system into multiple AGs, the system can distribute tasks such as file creation, deletion, and block allocation across different AGs, allowing multiple processes or nodes to access the file system simultaneously and avoiding excessive competition for a single centralized metadata or block resource. Furthermore, AGs improve fault isolation: damage or failure of a single AG typically does not affect the integrity of other AGs.

[0032] ZooKeeper is a distributed coordination service designed to provide efficient and reliable distributed synchronization, configuration management, naming services, cluster node management, and other basic support capabilities for distributed systems, helping to simplify the implementation of complex coordination logic in distributed applications.

[0033] In clustered file systems based on shared disk architectures, GFS2 and OCFS2 are typical examples. Both rely on distributed lock mechanisms to control data consistency between nodes, leading to significant system defects. For GFS2, its distributed locks rely on both network heartbeats and disk heartbeats to maintain availability. The upper layer provides range lock semantics through the glock state machine, which not only makes the lock management logic highly complex, increasing the difficulty of system maintenance and the cost of troubleshooting, but also generates a lot of overhead in small file read / write scenarios, resulting in a significant performance drop and failing to meet the needs of small file-intensive businesses. Although OCFS2 adds range locks on top of file-level locks to optimize lock granularity, it further exacerbates the complexity of lock management logic and also faces the problem of poor performance in small file read / write, making it difficult to adapt to application scenarios with high response speed requirements.

[0034] In distributed file systems with non-shared disk architectures, Lustre is a typical example. It employs a division of labor between a metadata server (MDS) and a data server (OSS). A single MDS centrally manages the file system's metadata (such as file paths, permissions, inode information, etc.). In this architecture, the MDS becomes the core bottleneck of system performance: when the file system size increases (e.g., the number of nodes increases, the number of files surges) or file operations become frequent (e.g., a large number of files are created, deleted, or renamed), the metadata processing capacity of the MDS cannot scale linearly, directly limiting the performance improvement of the entire file system. This results in insufficient system scalability, making it difficult to support ultra-large-scale data storage and high-concurrency access scenarios.

[0035] In summary, the relevant cluster file systems suffer from technical problems such as relying on DLM distributed locks, leading to complex logic and poor performance for small file read / write operations, or performance bottlenecks due to a single MDS architecture. They cannot meet the comprehensive requirements of current large-scale data storage scenarios for high performance, low complexity, and high scalability.

[0036] Therefore, to solve all or part of the above-mentioned technical problems, this application provides a method for processing document requests. To enable those skilled in the art to better understand the solution of this application, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments.

[0037] The execution of the file request processing method provided in this application embodiment relies on a distributed cluster file system architecture and has clear requirements for the collaborative design of hardware resources and software modules. Its specific application environment architecture and hardware architecture can be specifically described as follows:

[0038] From the perspective of application environment architecture, the core is a three-layer architecture consisting of a multi-node cluster, a global coordination layer, and a distributed storage layer.

[0039] First, a multi-node cluster consists of multiple file ownership nodes and at least one monitoring and coordination node or cluster management service. The file ownership nodes are responsible for actually processing file requests, managing local locks, and managing allocation group data. The monitoring and coordination node needs to maintain a global view and synchronize node status in real time through a heartbeat detection mechanism, while also supporting the calculation and updating of hash ring mapping rules; the global view stores node online status information, logical volume distribution information, and allocation group status information.

[0040] Secondly, the global coordination layer needs to provide a unified request access interface, path resolution module, and metadata query module to ensure efficient information parsing before request routing. The request access interface is used to receive client file requests; the path resolution module is used to extract mount points and map logical volume identifiers from file paths; and the metadata query module is used to call file system interfaces to obtain inode numbers and calculate allocation group identifiers.

[0041] Finally, the distributed storage layer needs to divide the backend storage into multiple logical volumes, each of which is further divided into independent allocation groups. It also supports precise mapping between device files and logical volumes through device mapping tables or storage management interfaces. At the same time, it configures independent metadata and data areas for each allocation group to ensure resource isolation at the allocation group level.

[0042] From a hardware architecture perspective, the following requirements must be met for clustered hardware deployment and performance adaptation: the file ownership node must have sufficient computing power and storage I / O performance to avoid I / O bottlenecks affecting request processing efficiency; the monitoring and coordination node must be configured with stable memory and network resources to ensure real-time updates of the global view and low latency of heartbeat detection; the backend storage hardware must support the partitioning of logical volumes and allocation groups, and must have data reliability assurance capabilities to prevent data loss in allocation groups from affecting services; in addition, the entire cluster must support low-latency communication between nodes to ensure efficient information synchronization during file request forwarding, heartbeat packet transmission, and disaster recovery switching, avoiding request routing errors or delayed fault recovery due to network latency.

[0043] Figure 1 A flowchart illustrating a file request processing method provided in this application embodiment is shown below. Figure 1 As shown, the method includes the following steps:

[0044] S101, Receive file request.

[0045] File requests are used to request operations on a target file, and these requests include the file path of the target file. File requests include, but are not limited to, requests to create, read, write, delete, and modify file attributes. They are used to request the creation, reading, writing, or deletion of a target file, or to modify the attributes of a target file.

[0046] S102. In response to a file request, obtain the inode identifier corresponding to the target file based on the file path.

[0047] The inode number of the target file indicates its offset within the target allocation group of the target logical volume. The inode number includes the identifier of the logical volume (LUN_ID), the identifier of the allocation group (AG_ID), and the file's physical location within the allocation group. The inode number directly determines the allocation group, and thus, based on the global view, the file's owning node. Each file has a unique inode number across the cluster.

[0048] This application divides the clustered file system into at least one logical volume, and then further divides each logical volume into multiple allocation groups. Different logical volumes are distinguished by a logical volume identifier (LUN_ID) for precise global location. Each logical volume is partitioned based on the characteristics of the storage device, the target file size, and the system's concurrency capabilities, and the size of each allocation group (AG) is set.

[0049] The allocation group partitioning strategy comprehensively considers the physical characteristics of the storage device (such as disk rotation speed, stripe size, and I / O performance), the size of the target file, and the system's concurrent processing capabilities (number of concurrent I / O requests, frequency of metadata operations, etc.). For example, in high-performance storage devices and large file scenarios, a larger AG size can be set to reduce metadata overhead; while in small file-intensive applications, a smaller AG size can be used to improve concurrency. After partitioning, each allocation group has an independent metadata area and data area, distinguished by a unique allocation group identifier (AG_ID), thus laying the foundation for subsequent AG-based file ownership node mapping and request routing.

[0050] For example, the cluster file system is divided into three logical volumes: LUN0, LUN1, and LUN2. LUN0 is further divided into 10 AGs, with allocation group identifiers AG0-AG9; LUN1 is divided into 10 AGs, with allocation group identifiers AG10-AG19; and LUN2 is divided into 10 AGs, with allocation group identifiers AG20-AG29. Each allocation group is 1GB in size. Assuming the logical volume identifier is LUN0 and the allocation group identifier is AG0, when the first file is created in allocation group AG0, if the inode offset number is 0, then the inode identifier of the first file is 000; when the second file is created in allocation group AG0, if the inode offset number is 1, then the inode identifier of the second file is 001.

[0051] When resolving the inode identifier corresponding to the target file based on the file path, the metadata of the target file is obtained using the interface provided by the cluster file system, starting from the file path. This metadata includes the inode number, which is a unique identifier for the file within its file system, but does not directly contain logical volume or allocation group information. Then, the mount point corresponding to the file path is resolved to find the device file corresponding to that mount point. This device file is then mapped to the backend logical volume identifier through a device mapping table or storage management interface. Further utilizing the file system's specific metadata structure, the target allocation group to which the target file belongs and the physical location of the target file within that group are calculated or queried to obtain the inode identifier.

[0052] By resolving the inode identifier from the file path, the storage location of a target file can be efficiently determined without relying on complex distributed lock mechanisms. First, the inode number from the file metadata is obtained using the cluster file system interface, ensuring the uniqueness of the file identifier. Then, the specific logical volume is located through the mount point and device mapping table, achieving a precise mapping from the file path to the backend storage device. Further, by combining this with the file system's specific metadata structure, the allocation group and its physical location of the file can be calculated or queried, thereby generating an inode identifier containing logical volume, allocation group, and offset information. This approach not only improves the accuracy and efficiency of file location but also lays the foundation for subsequent request routing based on allocation groups. This allows the system to quickly forward file requests to the corresponding home node for processing, reducing the overhead of cross-node coordination, simplifying lock management logic, and improving overall performance and scalability.

[0053] In some embodiments, after obtaining the inode identifier corresponding to the target file, the identifier AG_ID of the target allocation group is separated from the inode identifier through bitwise operations or preset mapping rules.

[0054] Bitwise operations are used to extract the bit field representing the AG_ID from the inode_ID of the target file's inode. For example, if the high 12 bits of the inode_ID represent the AG_ID and the low 20 bits represent the inode's index within that group, the system can use a right shift operation (inode_id >> 20) or a bitwise AND operation (inode_id & AG_MASK) to obtain the AG_ID. Another approach is to use a pre-defined mapping rule, such as a hash function or lookup table, to map the inode_ID to the corresponding AG_ID. This method is more flexible and can dynamically adjust the mapping relationship based on the storage layout. This method allows for quick determination of the allocation group where the file belongs, providing crucial information for subsequent home node lookup and request routing.

[0055] S103. Based on the global view and the identifier of the target allocation group, determine the first file ownership node corresponding to the target allocation group.

[0056] The global view contains the correspondence between each allocation group and multiple file ownership nodes. Each file ownership node has management authority over its assigned allocation group and can independently handle file requests without complex lock interactions with other nodes. It is important to emphasize that each allocation group is managed by only one file ownership node at any given time; there is no scenario where multiple file ownership nodes simultaneously operate on files within the same allocation group.

[0057] This application maintains a global view using ZooKeeper. The global view includes at least one of the following: node online status information, logical volume distribution information, and allocation group status information. Specifically, node online status information reflects whether the node is operating normally; logical volume distribution information determines the physical location of the logical volume within the cluster; and allocation group status information identifies whether the allocation group is in normal working order, whether data migration is underway, etc.

[0058] This global view integrates core information such as node online status, logical volume distribution, and allocation group status, providing unified data support for the efficient operation and stable management of the cluster file system. Specifically, node online status information allows the system to monitor the operational status of each file's owner node in real time, avoiding forwarding file requests to offline or faulty nodes, reducing request failure rates, and providing accurate basis for allocation group reallocation when nodes go offline, ensuring service continuity. Logical volume distribution information can quickly identify the physical storage location of logical volumes in the cluster. Combined with the previous mapping process from file path to logical volume, it further shortens the file location link, reduces the time overhead of cross-node queries, and improves request response speed. Allocation group status information can clearly identify whether the allocation group is working normally or in a special state such as data migration. This avoids data inconsistency or operation blocking caused by sending requests to allocation groups in abnormal states, and also provides real-time data reference for dynamic resource adjustments in the cluster (such as load balancing and allocation group migration during fault recovery), ensuring the rationality and accuracy of resource scheduling.

[0059] The construction of a global view provides a unified and real-time basis for decision-making in core processes such as request routing, fault handling, and resource scheduling of the cluster file system, effectively simplifying system management logic and improving the stability, reliability, and efficiency of cluster operation.

[0060] Each file system node in the cluster caches a global view locally, which reduces the number of accesses to ZooKeeper and improves the processing speed of file requests.

[0061] In some embodiments, the identifier of the target allocation group and the identifiers of multiple file ownership nodes are mapped onto a hash ring, and the distance between the identifier of the target allocation group and the identifiers of the multiple file ownership nodes is calculated on the hash ring. If the distance is less than or equal to a preset distance, the file ownership node corresponding to that distance is taken as the first file ownership node.

[0062] The above embodiment first maps the identifier of the target allocation group and the identifiers of multiple file ownership nodes onto a hash ring. Specifically, a hash function maps the identifier of the target allocation group to a position on the hash ring, and the identifier of each file ownership node to a 32-bit position on the hash ring, making the calculation of ownership relationships intuitive and efficient. The distance between the identifier of the target allocation group and the identifiers of each file ownership node is calculated on the hash ring, typically expressed as a clockwise step size or numerical difference on the ring. After calculating the distance between the identifier of the target allocation group and the identifier of any file ownership node, it is compared whether this distance is less than or equal to a preset distance. If so, it indicates that the file ownership node is the master node responsible for managing the target allocation group, and the file ownership node corresponding to this distance is determined as the file ownership node corresponding to the target allocation group, i.e., the first file ownership node. This not only allows for quick location of the target node but also minimizes the range of allocation groups affected when nodes join or leave the cluster, improving system stability and scalability.

[0063] By calculating the distance between the allocation group and each node on the ring and introducing a preset distance threshold as a criterion, the master node responsible for the allocation group (the node to which the first file belongs) can be quickly selected. Due to the characteristics of the hash ring, when a node joins or leaves, only the affiliation of some allocation groups needs to be adjusted, avoiding fluctuations caused by global remapping and improving the scalability and stability of the cluster. At the same time, the fixed master node affiliation strategy ensures that all operations on the same allocation group are routed to the same node, thus allowing local locks to replace complex distributed locks, simplifying lock management logic, reducing the overhead of cross-node coordination, and enhancing the system's concurrent processing capabilities and data consistency guarantees.

[0064] Optionally, for the target allocation group identifier, the file ownership node closest to its mapping position is found clockwise on the hash ring, and this node is taken as the ownership node of the target allocation group.

[0065] For example, suppose the cluster includes three nodes: node A, node B, and node C. After hash mapping, node A's position on the hash ring is 100, node B's position is 200, and node C's position is 300. Allocation group AG0 is mapped to position 50 on the hash ring, and allocation group AG10 is mapped to position 150. Distance calculation determines that node A is closest to allocation group AG0, and allocation group AG0 belongs to node A; similarly, node B is closest to allocation group AG10, and allocation group AG10 belongs to node B.

[0066] The clockwise search rule clarifies the search direction, eliminating the need to traverse all nodes. It only requires locating the nearest node along a fixed direction from the allocation group mapping position, shortening the search time for the belonging node. This is especially beneficial in clusters with a large number of nodes, reducing the computational and time overhead of the matching process and improving the response speed of request routing. Since allocation groups are uniformly mapped onto the ring using a hash function, combined with the nearest node allocation rule, the number of allocation groups on each file's belonging node is relatively balanced, effectively avoiding overload problems caused by too many allocation groups on a single node, and improving the overall resource utilization and concurrent processing capabilities of the cluster. More importantly, this mechanism ensures that the same allocation group always stably belongs to the same node (unless that node fails). All requests for files in that allocation group are routed to the same fixed node, laying the foundation for replacing complex distributed locks with node-local locks, further simplifying lock management logic, reducing the overhead of cross-node data consistency coordination, and balancing system performance and data reliability.

[0067] In some embodiments, after determining the first file-owning node corresponding to the target allocation group, a heartbeat detection mechanism is used to determine whether the first file-owning node is offline. If the first file-owning node is offline, the allocation groups of the first file-owning node are assigned to second file-owning nodes according to the global view, and the second file-owning node is not offline. Then, the previously received file request is sent to the second file-owning node for processing. This ensures that the file request can be correctly routed to the new second file-owning node for processing.

[0068] The global view contains the correspondence between the various allocation groups of the first file's owner node and the second file's owner node. It can be understood that the various allocation groups of the first file's owner node include the target allocation group.

[0069] After determining the primary file-owning node corresponding to the target allocation group, the system continuously monitors the node's online status through a heartbeat detection mechanism. Heartbeat detection can be implemented by periodically sending probe packets or exchanging status information. For example, the master node or monitoring service sends a heartbeat request to the primary file-owning node at fixed intervals and waits for a response. If no normal response is received within a preset timeout period, or if multiple heartbeats fail consecutively, the node is determined to be offline. Once a node is detected as offline, the system triggers a disaster recovery switchover process, removing all allocation groups (including the target allocation group) originally managed by that node from the global view. These allocation groups are then reassigned to other available nodes in the cluster, such as the secondary file-owning node, according to preset mapping rules or by re-performing hash calculations.

[0070] The aforementioned heartbeat mechanism, which involves periodically sending probe packets or exchanging status information, allows the system to accurately and in real-time detect the online status of the primary file-owning node. This prevents requests from being routed to invalid nodes due to node quiescent failures, thus reducing the file request failure rate. Simultaneously, the dual criteria of multiple heartbeat failures and timeout determination effectively filter out false positives caused by temporary anomalies such as network jitter, improving the accuracy of node offline detection. Once a node is confirmed offline, the system immediately triggers a disaster recovery switch. By removing the allocation groups associated with that node from the global view and quickly migrating these allocation groups to the secondary file-owning node based on preset rules or rehashing, the entire process requires no manual intervention, shortening fault recovery time, minimizing service interruption windows, and ensuring the continuity of user file operations.

[0071] Optionally, the distance between the identifier of the second file owner node and the identifier of the target allocation group on the hash ring is less than or equal to a preset distance. If the first file owner node is offline, the algorithm uses consistent hashing to calculate which file owner nodes(s) the allocation groups owned by the first file owner node need to be reassigned to.

[0072] For example, if the file owner node B corresponding to allocation group AG5 is offline, the allocation is reassigned. For allocation groups AG10-AG19 belonging to node B, the owner node of these allocation groups is recalculated according to the consistent hashing algorithm.

[0073] The overall migration strategy for allocation groups ensures that the ownership of all file resources under the original node is fully taken over, avoiding data chaos that may be caused by fragmented management of allocation groups. Meanwhile, the allocation method based on preset rules or rehashing ensures load balancing of the new ownership nodes, preventing a single node from becoming overloaded due to suddenly taking over a large number of allocation groups, further maintaining the overall stability of the cluster. In the event of node failure, a standardized disaster recovery process quickly restores ownership relationships, continuing the core advantage of fixed ownership nodes in simplifying lock management, while also mitigating the ownership invalidation problem that may occur due to node failure, allowing the entire cluster to maintain a highly efficient and reliable operating state even when facing node anomalies.

[0074] Based on the above embodiments, the first file ownership node is offline and has already assigned each allocation group of the first file ownership node to the second file ownership node. Then, according to the correspondence between the target allocation group and the second file ownership node, the global view is updated. The updated global view is pushed to other file ownership nodes so that they can synchronously update their locally cached global views. These other file ownership nodes are all file ownership nodes except the second file ownership node, including the first file ownership node.

[0075] It's understandable that if the allocation group of the first file owner node has been reassigned to another second file owner node, the mapping relationship between the file owner node and its allocation group has changed. Therefore, the global view containing these mapping relationships needs to be updated. Timely updating the global view cached locally on the node ensures that the global view used by the node is consistent with the actual state of the cluster.

[0076] The above update updates the mapping between allocation groups and file ownership nodes in the global view and notifies other nodes in the cluster to ensure that subsequent file requests can be correctly routed to the new ownership node, thereby ensuring service continuity and data consistency.

[0077] S104. Forward the file request to the first file owner node so that the first file owner node can process the file request.

[0078] When any node receives a file request, it resolves the inode identifier by resolving the file path of the target file, then locates the first file-owning node through the global view and forwards the request. There is no need to initiate a distributed lock request, maintain a lock state machine, or handle lock conflicts.

[0079] For example, suppose node A receives a file request containing the path to the requested file. Path resolution yields the file's inode identifier (053), indicating the file belongs to logical volume LUN0, allocation group AG5, and its index offset within allocation group AG5 is 3. Further, based on the global view, node A queries the file owner node corresponding to allocation group AG5, assuming it is node B. Node A forwards the file request to node B. Node B, as the file owner node for allocation group AG5, processes the file request, reads the file, and returns it to node A, which then returns it to the requester.

[0080] After forwarding the file request to the first file owner node, the first file owner node can directly access the small file data and metadata stored locally, avoiding the delays caused by lock waiting and lock release in distributed lock technology.

[0081] In summary, the file request processing method provided in this application has the following technical effects compared to related technologies:

[0082] Traditional shared disk architectures (such as GFS2 and OCFS2) require DLM distributed locks to ensure data consistency between nodes, and the addition of glock state machines and range locks further complicates lock management logic. This application, however, avoids this dependency on distributed locks at its source by establishing a fixed mapping between allocation groups and nodes. Specifically, a global view predefines the correspondence between each allocation group and node. Each allocation group is managed by a single file-owning node at any given time, eliminating the scenario of multiple nodes simultaneously operating on files within the same allocation group. When any node receives a file request, it only needs to resolve the inode identifier through the file path, extract the target allocation group identifier, locate the owning node through the global view, and forward the request. There is no need to initiate a distributed lock request, maintain a lock state machine, or handle lock conflicts. The entire process does not rely on network heartbeats or disk heartbeats to maintain lock availability, nor does it require complex lock granularity control logic, simplifying the system architecture and management process, and reducing maintenance difficulty and troubleshooting costs.

[0083] In DLM-dependent systems, frequent lock acquisition / release during small file read / write operations incurs significant overhead, leading to performance degradation. This application eliminates this additional performance overhead by directly processing the file attribution node. The inode identifier for small files also contains target allocation group information, allowing direct location of the unique attribution node through a global view, eliminating the need for lock negotiation or data synchronization across multiple nodes. After forwarding the request, the attribution node can directly access the locally stored small file data and metadata, avoiding latency issues associated with distributed locks (such as lock waiting and lock release confirmation). For batch small file requests, each small file can be distributed across different nodes for parallel processing based on its allocation group affiliation, rather than being centrally queued through a lock mechanism. This further improves the processing efficiency of small file-intensive tasks and adapts to application scenarios requiring high response speeds.

[0084] Traditional non-shared disk architectures (such as Lustre) cannot handle large-scale, high-concurrency scenarios due to centralized metadata management by a single MDS. This application achieves linear performance scaling through distributed metadata management. It eliminates the need for a separate MDS; file metadata (such as inode information and path relationships) is distributed across the file's owning nodes according to the allocation group. Each file's owning node manages only the metadata corresponding to its own allocation group, rather than having it centrally handled by a single node. When the system scales up (the number of nodes / files increases), metadata processing capabilities can be expanded synchronously by adding new allocation groups and mapping them to new nodes, avoiding the performance limitations of a single MDS. During high-concurrency file operations (such as creating or deleting a large number of files), requests are distributed to different nodes according to the allocation group, and each node processes its own metadata operations in parallel, preventing single-node overload and effectively supporting ultra-large-scale data storage and high-concurrency access scenarios.

[0085] like Figure 2 As shown in the embodiment of this application, another method for processing file requests is provided. This method is applied to the file ownership node and includes the following steps:

[0086] S201, Receive file request.

[0087] File requests are used to request operations on a target file, and these requests include the file path of the target file. File requests include, but are not limited to, requests to create, read, write, delete, and modify file attributes. They are used to request the creation, reading, writing, or deletion of a target file, or to modify the attributes of a target file.

[0088] S202. In response to a file request, the inode identifier corresponding to the target file is obtained by resolving the file path.

[0089] The inode number of the target file indicates its offset within the target allocation group of the target logical volume. The inode number includes the identifier of the logical volume (LUN_ID), the identifier of the allocation group (AG_ID), and the file's physical location within the allocation group. The inode number directly determines the allocation group, and thus, based on the global view, the file's owning node. Each file has a unique inode number across the cluster.

[0090] When resolving the inode identifier corresponding to the target file based on the file path, the metadata of the target file is obtained using the interface provided by the cluster file system, starting from the file path. This metadata includes the inode number, which is a unique identifier for the file within its file system, but does not directly contain logical volume or allocation group information. Then, the mount point corresponding to the file path is resolved to find the device file corresponding to that mount point. This device file is then mapped to the backend logical volume identifier through a device mapping table or storage management interface. Further utilizing the file system's specific metadata structure, the target allocation group to which the target file belongs and the physical location of the target file within that group are calculated or queried to obtain the inode identifier.

[0091] In some embodiments, after obtaining the inode identifier corresponding to the target file, bitwise operations are used to extract the bit field representing AG_ID from the inode_ID of the target file's inode. For example, if the high 12 bits of inode_ID represent AG_ID and the low 20 bits represent the index of the inode within that group, the system can use a right shift operation (inode_id >> 20) or a bitwise AND operation (inode_id & AG_MASK) to obtain AG_ID. Another approach is to use a preset mapping rule, such as a hash function or a lookup table, to map inode_ID to the corresponding AG_ID.

[0092] S203. Access the target file in local storage based on the inode identifier, and perform the corresponding operation for the file request on the target file.

[0093] For example, if the inode identifier is 001, it points to the second file created by allocation group AG0.

[0094] S204. Return the execution result to the sender of the file request.

[0095] If a file request is used to request modification of the attributes of a target file, then the target file with the modified attributes will be returned to the sender of the file request.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0097] like Figure 3 As shown, Figure 3 A document request processing apparatus provided in this application embodiment includes:

[0098] The receiving module 301 is used to receive a file request, which requests the target file to be operated on and contains the file path of the target file.

[0099] The parsing module 302 is used to respond to a file request by parsing the file path to obtain the inode identifier corresponding to the target file. The inode identifier is used to indicate the offset position of the target file on the target allocation group in the target logical volume.

[0100] The determination module 303 is used to determine the first file ownership node corresponding to the target allocation group based on the global view and the identifier of the target allocation group; wherein, the global view contains the correspondence between each allocation group and multiple file ownership nodes;

[0101] The sending module 304 is used to forward the file request to the first file owner node so that the first file owner node can process the file request.

[0102] As an optional implementation provided in this application, the determining module 303 is specifically used to: map the identifier of the target allocation group and the identifiers of multiple file ownership nodes onto a hash ring; calculate the distance between the identifier of the target allocation group and the identifiers of multiple file ownership nodes on the hash ring; and if the distance is less than or equal to a preset distance, take the file ownership node corresponding to the distance as the first file ownership node.

[0103] As an optional implementation provided in this application embodiment, the determining module 303 is further configured to: determine whether the first file ownership node is offline through a heartbeat detection mechanism; if the first file ownership node is offline, allocate each allocation group of the first file ownership node to the second file ownership node according to the global view; if the second file ownership node is not offline, send a file request to the second file ownership node so that the second file ownership node can process the file request.

[0104] As an optional implementation provided in this application, the device further includes an update module, configured to: update the global view according to the correspondence between the target allocation group and the second file ownership node; push the updated global view to other file ownership nodes so that the other file ownership nodes can synchronously update their locally cached global view; wherein, the other file ownership nodes are file ownership nodes other than the second file ownership node among the plurality of file ownership nodes, including the first file ownership node.

[0105] As an optional implementation provided in this application, the device further includes a setting module for setting the size of each allocation group according to the characteristics of the storage device, the size of the target file, and the system concurrency capability.

[0106] As an optional implementation provided in this application, the global view further includes at least one of the following: node online status information reflecting whether a node is operating normally; logical volume distribution information for determining the physical location of a logical volume in the cluster; and allocation group status information for determining whether an allocation group is operating normally and / or is undergoing data migration.

[0107] As an optional implementation provided in this application embodiment, the parsing module 302 is specifically used for: obtaining the metadata of the target file according to the file path; parsing the mount point corresponding to the file path to determine the device file corresponding to the mount point; mapping the device file through a device mapping table or storage management interface to obtain the identifier of the target logical volume; calculating the target allocation group to which the target file belongs and the offset position of the target file in the target allocation group according to the metadata and the identifier of the target logical volume; and obtaining the inode identifier according to the identifier of the target logical volume, the target allocation group and the offset position.

[0108] As an optional implementation provided in this application, the parsing module 302 is further configured to: separate the identifier of the target allocation group from the inode identifier through bitwise operations or preset mapping rules.

[0109] For a description of the features in the embodiment corresponding to the file request processing device, please refer to the relevant description in the embodiment corresponding to the file request processing method, which will not be repeated here.

[0110] This application embodiment also provides another document request processing apparatus, including:

[0111] A receiving module is configured to receive a file request, wherein the file request requests operation on a target file and the file request includes the file path of the target file;

[0112] The parsing module is used to respond to the file request by parsing the file path to obtain the inode identifier corresponding to the target file, wherein the inode identifier is used to indicate the offset position of the target file on the target allocation group in the target logical volume;

[0113] An execution module is used to access the target file stored locally based on the inode identifier and to perform the corresponding operation of the file request;

[0114] The sending module is used to return the processed target file to the sender of the file request.

[0115] like Figure 4 As shown, embodiments of this application also provide an electronic device, including a memory 401 and a processor 402. The memory 401 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above-described file request processing method embodiments.

[0116] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described file request processing method embodiments when running.

[0117] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0118] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described file request processing method embodiments.

[0119] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described file request processing method embodiments.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The foregoing has provided a detailed description of a document request processing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing file requests, characterized in that, include: Receive a file request, the file request being used to request operation on a target file, the file request containing the file path of the target file; In response to the file request, the inode identifier corresponding to the target file is obtained by parsing the file path. The inode identifier is used to indicate the offset position of the target file on the target allocation group in the target logical volume. The inode identifier includes the identifier of the logical volume where the file is located, the identifier of the allocation group where the file is located, and the physical location of the file in the allocation group. Based on the global view and the identifier of the target allocation group, the first file ownership node corresponding to the target allocation group is determined; wherein, the global view contains the correspondence between each allocation group and multiple file ownership nodes; The file request is forwarded to the first file ownership node so that the first file ownership node can process the file request; The step of responding to the file request and resolving the inode identifier corresponding to the target file based on the file path includes: obtaining the metadata of the target file based on the file path; resolving the mount point corresponding to the file path to determine the device file corresponding to the mount point; mapping the device file through a device mapping table or storage management interface to obtain the identifier of the target logical volume; calculating the target allocation group to which the target file belongs and the offset position of the target file in the target allocation group based on the metadata and the identifier of the target logical volume; and obtaining the inode identifier based on the identifier of the target logical volume, the target allocation group, and the offset position.

2. The method according to claim 1, characterized in that, The step of determining the first file ownership node corresponding to the target allocation group based on the global view and the identifier of the target allocation group includes: Map the identifier of the target allocation group and the identifiers of the multiple file ownership nodes onto a hash ring; On the hash ring, calculate the distance between the identifier of the target allocation group and the identifiers of the plurality of file ownership nodes; If the distance is less than or equal to a preset distance, the file ownership node corresponding to the distance is taken as the first file ownership node.

3. The method according to claim 1, characterized in that, After determining the first file ownership node corresponding to the target allocation group based on the global view and the identifier of the target allocation group, the method further includes: The heartbeat detection mechanism is used to determine whether the node to which the first file belongs is offline. If the first file ownership node is offline, then according to the global view, each allocation group of the first file ownership node is assigned to the second file ownership node; the second file ownership node is not offline. A file request is sent to the second file ownership node for processing.

4. The method according to claim 3, characterized in that, The method further includes: Update the global view based on the correspondence between the target allocation group and the second file ownership node; The updated global view is pushed to other file ownership nodes so that the other file ownership nodes can synchronously update the global view cached locally. The other file ownership nodes are the file ownership nodes other than the second file ownership node among the plurality of file ownership nodes, including the first file ownership node.

5. The method according to claim 1, characterized in that, The method further includes setting the size of each allocation group according to the characteristics of the storage device, the size of the target file, and the system's concurrency capabilities.

6. The method according to claim 1, characterized in that, The global view also includes at least one of the following: Node online status information used to reflect whether a node is operating normally; Logical volume distribution information used to determine the physical location of logical volumes within the cluster; Allocation group status information used to determine whether the allocation group is working properly and / or whether it is undergoing data migration.

7. The method according to claim 1, characterized in that, After responding to the file request and obtaining the inode identifier corresponding to the target file based on the file path parsing, and before determining the first file ownership node corresponding to the target allocation group based on the global view and the identifier of the target allocation group, the method further includes: The identifier of the target allocation group is separated from the inode identifier by bitwise operations or preset mapping rules.

8. A method for processing file requests, characterized in that, Applied to the file ownership node, the method includes: Receive a file request, the file request being used to request operation on a target file, the file request containing the file path of the target file; In response to the file request, the inode identifier corresponding to the target file is obtained by parsing the file path. The inode identifier is used to indicate the offset position of the target file on the target allocation group in the target logical volume. The inode identifier includes the identifier of the logical volume where the file is located, the identifier of the allocation group where the file is located, and the physical location of the file in the allocation group. Access the target file stored locally based on the inode identifier, and perform the corresponding operation of the file request on the target file; The execution result is returned to the sender of the file request; The step of responding to the file request and resolving the inode identifier corresponding to the target file based on the file path includes: obtaining the metadata of the target file based on the file path; resolving the mount point corresponding to the file path to determine the device file corresponding to the mount point; mapping the device file through a device mapping table or storage management interface to obtain the identifier of the target logical volume; calculating the target allocation group to which the target file belongs and the offset position of the target file in the target allocation group based on the metadata and the identifier of the target logical volume; and obtaining the inode identifier based on the identifier of the target logical volume, the target allocation group, and the offset position.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the file request processing method as described in any one of claims 1 to 8 when executing the computer program.

Images