File operation method and device, computer device, and storage medium

Abstract

The application relates to a file operation method and device, computer equipment and a storage medium. The method comprises the following steps: in response to a file operation request, determining a file path of a target file specified by the file operation request; searching a file directory to which the target file belongs step by step from a starting point to an ending point along the file path; in the process of searching the file directory to which the target file belongs step by step, sequentially performing locking processing on the file directory searched step by step and performing locking processing on the target file; performing file operation on the target file; wherein in the process of performing write operation on the target file, other file operation requests of the target file are in a waiting state, so that the target file can be prevented from being modified. The method can improve the operation efficiency of data in a distributed computing platform.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a file operation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the improvement of computer storage capacity and the development of complex algorithms, the amount of data has grown exponentially in recent years. As a result, some distributed computing platforms have emerged to enable distributed computing of massive amounts of data in clusters composed of a large number of computers, providing effective support for the distributed storage and computing of big data.

[0003] In traditional solutions for data operations within distributed computing platforms, a global lock exists within the platform that governs the directory node tree. All data operations involving data within this tree must acquire this global lock before proceeding. However, if another data operation requests to acquire the lock, that request will be blocked until the lock is released, severely hindering the efficiency of data operations within the distributed computing platform. Summary of the Invention

[0004] Therefore, it is necessary to provide a file operation method, apparatus, computer device, and storage medium that can improve the efficiency of data operation in a distributed computing platform, addressing the aforementioned technical problems.

[0005] A file operation method, the method comprising:

[0006] In response to a file operation request, determine the file path of the target file specified in the file operation request;

[0007] The file path is searched level by level from the starting point to the ending point to find the file directory to which the target file belongs;

[0008] During the process of searching the file directory to which the target file belongs, the file directories found at each level are locked sequentially, as are the target file itself.

[0009] File operations are performed on the target file; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0010] A file manipulation device, the device comprising:

[0011] The determination module is used to determine the file path of the target file specified in the file operation request in response to the file operation request;

[0012] The search module is used to search for the file directory to which the target file belongs, level by level, from the start point to the end point of the file path;

[0013] The locking module is used to lock the file directories found level by level and the target file in turn during the process of searching the file directories to which the target file belongs.

[0014] An operation module is used to perform file operations on the target file; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0015] In one embodiment, the file operation request includes a file read request;

[0016] The locking module is further configured to obtain shared locks from the lock resource pool or create the shared locks; lock the file directories found level by level based on different shared locks; and lock the target file according to different shared locks.

[0017] In one embodiment, the file operation request includes a file write request;

[0018] The locking module is further configured to obtain shared locks and exclusion locks from the lock resource pool, or create the shared locks and exclusion locks; lock the file directories found level by level based on different shared locks; and lock the target file based on the exclusion locks.

[0019] In one embodiment, the file operation request includes a file creation request;

[0020] The locking module is further configured to acquire shared locks and exclusion locks from the lock resource pool, or create the shared locks and exclusion locks; when the file directory found level by level is not the parent node of the target file, the found file directory is locked based on different shared locks; when the file directory found level by level is the parent node of the target file, the found file directory is locked based on the exclusion lock; and the target file is locked based on different exclusion locks.

[0021] In one embodiment, the shared lock and the exclusive lock are different lock objects that include a reference counter; the device further includes:

[0022] An adjustment module is used to adjust the reference counters in the released shared lock and the released exclusive lock respectively when the shared lock and the exclusive lock are released, so that the count in the adjusted reference counter is used to indicate that the released shared lock and the exclusive lock are not occupied;

[0023] The storage module is used to store the adjusted reference counter count, as well as the lock identifiers of the shared lock and the exclusion lock, in the lock recycling pool.

[0024] In one embodiment, the device further includes:

[0025] The generation module is used to generate lock timeout monitoring threads;

[0026] The monitoring module is used to monitor the occupancy time of locks in the lock resource pool through the lock timeout monitoring thread;

[0027] The writing module is used to write the lock information of the lock into the log through the lock timeout monitoring thread when the lock occupancy time reaches the time threshold.

[0028] The display module is used to display the lock information in the log through the lock view when an abnormality occurs in the lock resource pool that is in an occupied state.

[0029] In one embodiment, the file operation request includes a first move request and a second move request to move the target file; the file path includes a first file path and a second file path corresponding to the first move request and the second move request, respectively;

[0030] The locking module is further configured to, when the path value of the first file path is less than the path value of the second file path, sequentially lock the file directories found level by level in the first file path and lock the target file, until the target file is operated on completely, then sequentially lock the file directories found level by level in the second file path and lock the target file after the operation is completed.

[0031] In one embodiment, the device further includes:

[0032] The locking module is also used to, during the process of reading the target file, if it receives other file read operation requests for the target file, lock the file directories found level by level based on the read locks in the lock resource pool, and lock the target file in turn.

[0033] The operation module is also used to perform file read operations on the target file.

[0034] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0035] In response to a file operation request, determine the file path of the target file specified in the file operation request;

[0036] The file path is searched level by level from the starting point to the ending point to find the file directory to which the target file belongs;

[0037] During the process of searching the file directory to which the target file belongs, the file directories found at each level are locked sequentially, as are the target file itself.

[0038] File operations are performed on the target file; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0039] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0040] In response to a file operation request, determine the file path of the target file specified in the file operation request;

[0041] The file path is searched level by level from the starting point to the ending point to find the file directory to which the target file belongs;

[0042] During the process of searching the file directory to which the target file belongs, the file directories found at each level are locked sequentially, as are the target file itself.

[0043] File operations are performed on the target file; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0044] The aforementioned file operation method, apparatus, computer equipment, and storage medium, in response to a file operation request, determine the file path of the target file specified in the file operation request; traverse the file path from its starting point to its ending point, level by level, searching for the file directory to which the target file belongs; during the level-by-level search of the file directory to which the target file belongs, locking is performed sequentially on the found file directories and on the target file itself, thereby achieving individual locking of the file directories and the target file corresponding to different nodes in the file path. This ensures that file operations on other different file paths are not affected during file operations on the target file, thus improving the data operation efficiency in the distributed computing platform and consequently increasing the throughput efficiency of the distributed computing platform. Furthermore, during the write operation on the target file, other file operation requests for the target file are in a waiting state, ensuring that the current write operation on the target file is not interrupted, which is beneficial for improving data accuracy. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the application environment of a file operation method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a file operation method in one embodiment;

[0047] Figure 3 This is a schematic diagram illustrating the recovery of shared locks and exclusion locks in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the steps of searching for and locking a file directory in a directory node tree in one embodiment.

[0049] Figure 5 This is a schematic diagram illustrating the structure of searching for a file directory in a directory node tree in one embodiment.

[0050] Figure 6 This is a schematic diagram illustrating the locking process for a file directory and a target file in one embodiment;

[0051] Figure 7 This is a schematic diagram illustrating the locking process for the file directory and the target file in another embodiment;

[0052] Figure 8 This is a schematic diagram illustrating the locking process for the file directory and the target file in another embodiment;

[0053] Figure 9 A schematic diagram illustrating the locking of a file directory and a target file under different operations in one embodiment;

[0054] Figure 10 This is a schematic diagram illustrating different locking methods in one embodiment;

[0055] Figure 11 This is a flowchart illustrating a file operation method in another embodiment;

[0056] Figure 12 This is a schematic diagram comparing the throughput of the Namenode of this application with the throughput of the native Namenode in one embodiment;

[0057] Figure 13 This is a structural block diagram of a file operation device in one embodiment;

[0058] Figure 14 This is a structural block diagram of the file operation device in another embodiment;

[0059] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The file manipulation methods provided in this application can be applied to, for example... Figure 1 The application environment shown includes terminals 102, 104, 106, and server 108. Terminal 102 sends a file operation request to server 108. In response to the file operation request, server 108 determines the file path of the target file specified in the file operation request; searches for the file directory to which the target file belongs level by level along the file path from the start to the end; during the process of searching for the file directory to which the target file belongs, it locks the file directories found level by level and locks the target file; and performs file operations on the target file. During the write operation on the target file, other file operation requests for the target file from terminals 104 and 106 are in a waiting state.

[0062] Among them, terminals 102, 104 and 106 can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these.

[0063] Server 108 can be a standalone physical server or a service node in a blockchain system. These service nodes form a peer-to-peer (P2P) network, where the P2P protocol is an application-layer protocol running on top of the Transmission Control Protocol (TCP). Server 108 can also house a distributed computing platform for distributed processing of large datasets; this platform can be a software system platform based on the Hadoop distributed system architecture.

[0064] In addition, server 108 can also be a server cluster consisting of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0065] Terminals 102, 104, and 106 and server 108 can be connected via wireless or wired network connection, and this application does not impose any restrictions on this.

[0066] In one embodiment, such as Figure 2 As shown, a file operation method is provided, which can be applied to... Figure 1 Taking server 108 as an example, the explanation includes the following steps:

[0067] S202, in response to a file operation request, determines the file path of the target file specified in the file operation request.

[0068] File operation requests can be Remote Procedure Call (RPC) requests from the target client to operate on the target file, including any of the following: file read requests, file write requests, file move requests, and file creation requests. Furthermore, file operation requests can be RPC requests from users of two clients to move the target file. The following sections will explain file read requests, file write requests, file move requests, and file creation requests in detail:

[0069] A file read request can be an RPC request used to read a target file, such as calling a read (READ) type function to read the target file. This read type function can be the getFileInfo function, listStatus function, and getBlockLocation function, etc.

[0070] A file write request can be a request to write data to a target file, such as calling a write (WRITE) type function to write data to the target file. This write type function can be a setTime function, a setOwner function, or a completeFile function, etc.

[0071] A file move request can be used to rename or move a target file. This could involve calling a rename function or a move function to rename or move the target file. The rename function could be the `rename` function, and the move function could be the `move` function. It's important to note that renaming a target file using a rename function is, at the computer's underlying implementation, equivalent to moving the target file. For example, if the original file path of target file `f` is ` / d1 / d2 / f`, and it is renamed to `f1`, then the file path of target file `f1` will be ` / d1 / d2 / f1`.

[0072] A file creation request can be a request to create a target file, such as a request to call a file creation class function to create a target file. This file creation class function can be the create function or the mkdir function, etc.

[0073] The target file can be the file to be operated on as specified in the file operation request. For example, if the client initiates a file operation request to operate on file a, then file a is the target file.

[0074] A file path can be the route through which folders (or directories) are traversed when searching for a target file on the disk. The target file can be found on the disk through this file path.

[0075] In one embodiment, the file operation request may carry the file path of the target file, or it may carry the file identifier of the target file. Therefore, the server can determine the file path of the target file based on the file operation request. Specifically, the server reads the file path of the target file from the file operation request; or, it reads the file identifier of the target file from the file operation request and finds the file path of the target file based on the file identifier.

[0076] S204: Search for the directory containing the target file level by level from the start to the end of the file path.

[0077] In this context, the starting point of the file path represents the root directory of the target file, and the ending point represents the target file. For example, in d0 / d1 / d2 / f, d0 represents the root directory and f represents the target file.

[0078] A file directory can be an index of a target file, that is, a mapping between the name of a target file and its physical location. This index of a target file is called a file directory. In practical applications, this file directory can specifically be a folder. File directories are divided into first-level directories, second-level directories, and multi-level directories. For multi-level directories, each disk has a root directory for different files. The root directory can contain one or more subdirectories and files. Subdirectories can contain files and can also contain subdirectories of the next level, and so on, such as d0 / d1 / d2 / f. d0 is the first node in the file path, used to represent the root directory; d1 is the second node in the file path, used to represent the first-level subdirectory (i.e., the folder named d1); d2 is the third node in the file path, used to represent the second-level subdirectory (i.e., the folder named d2); and f is the last node in the file path, used to represent the target file.

[0079] In one embodiment, the file path includes multiple path nodes. The server starts from the first path node in the file path and searches for the file directory to which the target file belongs, level by level, along the file path to the endpoint. For example, for the first path node, the server searches for the root directory containing the target file, or the root directory containing a file subdirectory, based on the node identifier of the first path node (e.g., d0 in d0 / d1 / d2 / f). After finding the root directory, the server searches for the file subdirectory containing the target file (i.e., the first-level subdirectory), or the file subdirectory containing a second-level subdirectory, based on the node identifier of the second path node (e.g., d1 in d0 / d1 / d2 / f). It should be noted that "directly storing the target file" can mean that the target file is directly stored in the current directory; conversely, "indirectly storing the target file" can mean that the current directory contains a next-level subdirectory, which directly or indirectly stores the target file.

[0080] S206, In the process of searching the file directory to which the target file belongs, lock the file directory found level by level and lock the target file in turn.

[0081] Locking can refer to using appropriate locks to lock file directories or target files in order to control access to file directories or target files by other clients or threads.

[0082] In one embodiment, during the process of searching for the file directory to which the target file belongs, the server locks the found file directory each time it finds a corresponding level of file directory, based on the file operation type corresponding to the file operation request. After the file directories corresponding to each path node in the file path have been locked, the target file is locked based on the file operation type corresponding to the file operation request.

[0083] Different types of file operation requests correspond to different file operation types. For example, a file read request corresponds to a file read operation (i.e., a read-type operation), a file write request corresponds to a file write operation (i.e., a write-type operation), a file move request corresponds to a file move operation, and a file creation request corresponds to a file creation operation. Furthermore, different file operation types require different locking mechanisms.

[0084] Specifically, during the process of searching for the directory containing the target file level by level, each time a directory that directly or indirectly contains the target file is found, the server acquires a lock of the corresponding type based on the file operation request, and then locks the found directory based on the acquired lock. After locking the directory corresponding to each path node in the file path, the server continues to search for the target file, acquires a lock of the corresponding type based on the file operation request, and then locks the target file based on the acquired lock.

[0085] For example, when the file operation type is a file read operation, the server uses read locks to lock both the found file directory and the target file; as another example, when the file operation type is a file write operation, the server uses read locks to lock the found file directory and write locks to lock the target file.

[0086] It should be noted that the read lock described above is a shared lock, meaning that different access objects can perform read operations on the locked file or directory. For example, when thread A or client A performs a read operation on a file or directory, thread B or client B can also perform a read operation on that file or directory. Furthermore, the write lock described above is an exclusive lock. When one access object performs a write operation on a locked file or directory, other access objects will not be able to perform write or read operations on the locked file or directory. For example, when thread A or client A performs a write operation on a file or directory, thread B or client B will not be able to perform other operations on that file or directory, such as read or write operations.

[0087] S208, Perform file operations on the target file; during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0088] In one embodiment, different types of file operation requests correspond to different types of file operations. When the file operation request is a file read request, a read operation is performed on the target file, i.e., reading the target file; when the file operation request is a file write request, a write operation is performed on the target file, such as writing new data into the target file; when the file operation request is a file move request, a move operation is performed on the target file, such as moving it from folder a to folder b; when the file operation request is a file creation request, the file operation is performed on the file directory corresponding to the last path node in the file path, the current file directory, and the target file.

[0089] In one embodiment, since write locks are exclusive locks, when a file operation request from another client is received during a write operation on the target file, the server can store the request in a waiting queue, thus placing it in a waiting state. Since read locks are shared locks, when a read operation on the target file is received, if another file read operation request (e.g., from another client) is received, the server uses the read locks in the lock resource pool to lock the found file directories level by level, and then locks the target file itself before performing the read operation on the target file.

[0090] In one embodiment, the shared lock and the exclusive lock are different lock objects that include reference counters. After S208, the server can reclaim the released shared lock and exclusive lock. The specific reclamation method includes: when the shared lock and the exclusive lock are released, the server adjusts the reference counters in the released shared lock and exclusive lock respectively, so that the adjusted reference counter count indicates that the released shared lock and exclusive lock are not occupied; the adjusted reference counter count, as well as the lock identifier of the shared lock and the exclusive lock, are stored in the lock reclamation pool. By reclaiming the released shared lock and exclusive lock, the lock utilization rate is improved, and memory garbage is reduced.

[0091] The server can create an asynchronous recycling thread. When the number of referenced locks (including shared and exclusive locks) in the lock resource pool reaches a reference threshold, the server can use this recycling thread to adjust the reference counters of the released locks. For example, a reference counter of 1 indicates that the lock is referenced, and 0 indicates that it is not referenced. In this case, the reference counter is adjusted from 1 to 0, thereby reclaiming the released locks back into the lock resource pool. Figure 3 As shown.

[0092] In the above embodiments, in response to a file operation request, the file path of the target file specified in the file operation request is determined; the file directory to which the target file belongs is searched level by level along the file path from the start point to the end point; during the process of searching the file directory to which the target file belongs, the file directories found level by level are locked sequentially, and the target file is also locked, thereby realizing the individual locking of the file directories and the target file corresponding to different nodes in the file path. This ensures that file operations on other different file paths are not affected during file operations on the target file, thus improving the data operation efficiency in the distributed computing platform and consequently increasing the throughput efficiency of the distributed computing platform. Furthermore, during the write operation on the target file, other file operation requests for the target file are in a waiting state, ensuring that the current write operation on the target file is not interrupted, which is beneficial to improving data accuracy.

[0093] In one embodiment, such as Figure 4 As shown, S204 may specifically include:

[0094] S402, obtain the directory node tree corresponding to the file operation request; the file path is a branch in the directory node tree that starts from the node corresponding to the root directory and ends at the node corresponding to the target file.

[0095] The directory node tree (also known as the Inode tree) is a structure tree within the distributed computing platform based on the Hedrup distributed system architecture. It is a structure tree composed of the identifiers corresponding to file directories and the identifiers corresponding to the files stored in those directories as nodes.

[0096] Furthermore, the directory node tree includes multiple branches, and the file path is one of these branches, specifically a branch that starts with the node corresponding to the root directory and ends with the node corresponding to the target file. For example... Figure 5 As shown, there are 3 branches in the directory node tree. The branch that includes the qq / , dn, and fn1 nodes is the file path of the target file. The file path is qq / dn / fn1, where the qq / node represents the root directory of the target file, and the fn1 node represents the target file.

[0097] In one embodiment, S402 may specifically include: when the file operation request carries the file path of the target file, the server reads the file path in the file operation request and searches for a matching directory node tree based on the file path; when the file operation request carries the file identifier of the target file, the server searches for a node that matches the file identifier based on the file identifier, thereby obtaining a directory node tree including the node.

[0098] S404: In the directory node tree, starting from the node corresponding to the root directory, search the file directories level by level along the file path to obtain the file directories containing the target file at each level.

[0099] For example, such as Figure 5 As shown, first, the root node (i.e., the node corresponding to the root directory) in the directory node tree is searched. Based on the identifier of the root node, the root directory of the target file fn1 is searched (i.e., the file directory to which the target file belongs and which is the file directory corresponding to the root node). Then, S406 is executed; when the locking is completed, ... Figure 5 The search continues in the direction of the file path, searching for the second node. Based on the identifier 'dn' of the second node, it locates the directory where the target file 'fn1' is located (a subdirectory of the root directory, i.e., the folder directly containing the target file 'fn1'), and then executes S406. It should be noted that... Figure 5 The locked dashed lock is a read lock (a shared lock), and the locked solid lock is a write lock (an exclusive lock).

[0100] Correspondingly, S206 may specifically include:

[0101] S406, during the process of searching the file directory to which the target file belongs, lock the file directories that include the target file in the search at each level.

[0102] S408 locks the target file.

[0103] The specific steps in S406 to S408 above can be found by referring to Figure 2 S206 in the embodiment. It should be noted that in this embodiment, only the file directories and target files corresponding to nodes in the file path within the directory node tree are locked; directories and files corresponding to nodes in other branches are not locked. Figure 5 As shown.

[0104] In the above embodiments, when a file operation request is received, only the directories and target files corresponding to the nodes in the file path within the directory node tree are locked, while the directories and files corresponding to the nodes in other branches are not locked. This ensures that the operations of other subdirectories under the same root directory in the directory node tree are not affected, avoiding the problem of reduced operation efficiency caused by locking the root directory of the entire target file in the distributed computing platform in the traditional solution, and effectively improving the efficiency of data operations in the distributed computing platform.

[0105] Different types of file operation requests require different lock types and locking strategies. For S206, three locking scenarios can be described as follows:

[0106] (1) Locking scenario under file read request, use shared lock to lock the file directory and target file.

[0107] In one embodiment, the file operation request includes a file read request; S206 may specifically include: the server acquiring shared locks from the lock resource pool or creating shared locks; locking the file directories found level by level based on different shared locks; and locking the target file according to different shared locks.

[0108] The lock pool stores several locks of different types. When locking is required, a lock of the appropriate type is retrieved from the lock pool. A shared lock refers to a lock that, when other file operation requests of the same type are received, can continue to lock the target file and its corresponding directory, thereby enabling file operations such as read operations on the target file and its corresponding directory.

[0109] After locating the file directory at the target level (such as the level where the root node is located), the server acquires an unreferenced (i.e., unoccupied) shared lock from the lock resource pool if one exists, and then locks the found file directory based on that shared lock. Next, it continues searching for file directories at child levels (i.e., sub-levels of the target level). If an unreferenced shared lock exists in the lock resource pool, it acquires that lock and locks the child level's file directory based on that shared lock, and so on, until the target file is locked using different shared locks. Figure 6 As shown.

[0110] Furthermore, after locating the file directory of the target level (such as the level where the root node is located), if all shared locks in the lock resource pool are referenced, the server creates a shared lock and then locks the file directory of the target level based on the created shared lock. Next, it continues to search for file directories in sub-levels (i.e., sub-levels of the target level), creating shared locks and locking the file directories of those sub-levels based on the created shared locks, and so on, until the target file is locked based on different created shared locks. It should be noted that if there are unreferenced shared locks in the lock resource pool, there is no need to create a new shared lock.

[0111] (2) Locking scenarios under file write requests: use shared locks to lock the file directory and use exclusive locks to lock the target file.

[0112] In one embodiment, the file operation request includes a file write request; S206 may specifically include: the server acquiring shared locks and exclusive locks from the lock resource pool, or creating shared locks and exclusive locks; locking the file directories found level by level based on different shared locks; and locking the target file based on the exclusive lock.

[0113] After locating the file directory at the target level (such as the level where the root node is located), the server acquires an unreferenced shared lock from the lock resource pool if one exists, and then locks the found file directory using that shared lock. Next, it continues searching for file directories at child levels (i.e., sub-levels of the target level). If an unreferenced shared lock exists in the lock resource pool, the server acquires that lock and locks the child level's file directory using that shared lock, and so on, until the target file is locked using an exclusive lock. Figure 7 As shown.

[0114] After locating the file directory of the target level (such as the level where the root node is located), if all shared locks in the lock resource pool are referenced, the server creates a shared lock and then locks the file directory of the target level based on the created shared lock. Next, it continues to search for file directories in child levels (i.e., sub-levels of the target level), creating shared locks and locking the file directories of those child levels based on the created shared locks, and so on, until the target file is locked based on an exclusive lock. It should be noted that if there are unreferenced shared locks in the lock resource pool, there is no need to create a new shared lock.

[0115] (3) In the case of locking under file creation request, the target file and its parent node corresponding file directory are locked by using an exclusion lock, and other file directories are locked by using a shared lock.

[0116] In one embodiment, the file operation request includes a file creation request; S206 may specifically include: the server acquiring shared locks and exclusive locks from the lock resource pool, or creating shared locks and exclusive locks; when the file directory found level by level is not the parent node of the target file, locking the found file directory based on different shared locks; when the file directory found level by level is the parent node of the target file, locking the found file directory based on exclusive locks; locking the target file based on different exclusive locks, such as... Figure 8 As shown.

[0117] The specific locking steps can be found in scenarios (1) and (2). For a more intuitive understanding of the differences between lock types and locking strategies in the three scenarios, please refer to... Figure 9 .

[0118] In one embodiment, the server can also monitor locks that have timed out. The specific steps include: the server generating a lock timeout monitoring thread; monitoring the duration of locks in the lock resource pool that are currently in use through the lock timeout monitoring thread; when the lock's duration reaches a threshold, writing the lock information to a log through the lock timeout monitoring thread; and displaying the lock information in the log through a lock view when an abnormality occurs in a lock in the lock resource pool. Therefore, when a lock malfunctions, such as a deadlock, the specific anomaly can be found in the logs, allowing for timely identification and resolution of the problem's source.

[0119] The lock timeout monitoring (monitorTimeoutLock) thread can be used to monitor the duration of lock occupancy, and when the occupancy duration reaches the time threshold, obtain the lock information, such as the thread identifier of the thread referencing the lock, the occupancy duration, and the operation when using the lock (such as file read operation), and then write the lock information to the log.

[0120] In the above embodiments, different types of locks and different locking strategies are used for different types of file operation requests. This ensures that while a write operation is being performed on the target file, other file operation requests for the target file are in a waiting state, guaranteeing that the current write operation is not interrupted and improving data accuracy. Furthermore, during a read operation on the target file, other file read requests can lock both the target file and its corresponding directory to ensure file sharing. Additionally, during the creation of a target file, since an exclusion lock is applied to the target file and its parent directory, other file operation requests targeting the target file and its parent directory are also in a waiting state, ensuring that the creation operation of the target file in the parent directory is not interrupted and that the target file is not affected by other requests, further improving data accuracy. Finally, each node in the directory node tree does not need to correspond to a single lock; instead, locks are acquired from a lock resource pool when needed, reducing the number of locks.

[0121] In one embodiment, the locking methods for the file directory and target file differ depending on the information carried in the file operation request. Furthermore, when operations involve multiple directories, the locking methods for the file directory and target file also differ. These are described in three cases:

[0122] Case 1: Lock the target file level by level from top to bottom according to its file path.

[0123] In one embodiment, the server reads the file path of the target file in the file operation request, and then searches for the file directory to which the target file belongs level by level along the file path from the start to the end. During the process of searching for the file directory to which the target file belongs, the server locks the file directories found level by level and locks the target file in turn. Then, the server performs file operations on the target file.

[0124] like Figure 10 As shown in Figure (a), the server reads the file path qq / d1 / d2 / f from the file operation request, and then searches for the corresponding folder and target file for each node sequentially from top to bottom, that is, it searches for the qq folder, d1 folder, d2 folder, and target file f in sequence. During the search process, when a folder or target file is found, it locks the found qq folder, d1 folder, d2 folder, or target file f. For example, if the file operation request is a file read operation, the read lock is used to lock the qq folder, d1 folder, d2 folder, and target file f in sequence; if the file operation request is a file write operation, the read lock is used to lock the qq folder, d1 folder, d2 folder, and target file f in sequence, and the write lock is used to lock the target file f.

[0125] Case 2: Lock the target file based on its file identifier.

[0126] In one embodiment, the server reads the file identifier of the target file in the file operation request, then obtains the file path of the target file based on the file identifier, and then searches for the file directory to which the target file belongs level by level along the file path from the start point to the end point; during the process of searching for the file directory to which the target file belongs, the server locks the file directories found level by level and locks the target file in turn; and then performs file operations on the target file.

[0127] like Figure 10 As shown in Figure (b), the server reads the file identifier (fileid) of the target file from the file operation request, then converts the fileid into the file path of the target file (qq / d1 / d2 / f), and then searches for the folder and target file corresponding to each node in a hierarchical manner from top to bottom, that is, it searches for the qq folder, d1 folder, d2 folder and target file f in sequence. During the hierarchical search, when a folder or target file is found, the found qq folder, d1 folder, d2 folder or target file f is locked.

[0128] Case 3: Locking based on multi-directory operations.

[0129] In one embodiment, the file operation request includes a first move request and a second move request to move the target file; the file path includes a first file path and a second file path corresponding to the first move request and the second move request, respectively; S206 may specifically include: when the path value of the first file path is less than the path value of the second file path, the server sequentially locks the file directories found level by level in the first file path and locks the target file until the target file is operated on completely, then sequentially locks the file directories found level by level in the second file path and locks the target file after the operation is completed.

[0130] Multi-directory operations can refer to two or more file move requests operating on the same target file. For example, thread A wants to move target file f from path1 to path2, while thread B wants to move target file f from path2 to path1.

[0131] For example, such as Figure 10 As shown in Figure (c), if thread A wants to move target file f from file path qq / d1 / d2 to qq / d3 / d4, and thread B wants to move target file f from file path qq / d3 / d4 to qq / d1 / d2, if the server does not compare the ASCII codes of these two paths, the following problem may occur: Thread A has already locked the directory corresponding to d2 using a write lock, and now requests to acquire a write lock to lock the directory corresponding to d4; correspondingly, thread B has already locked the directory corresponding to d4 using a write lock, and now requests to acquire a write lock to lock the directory corresponding to d2. This will lead to a deadlock, where both thread A and thread B hold the lock that the other wants and are waiting for the other to release the write lock. Therefore, the server compares the ASCII codes of the initial file paths of the two threads, and then determines the locking order based on the size of the ASCII codes. For example, if the initial file path of thread A is qq / d1 / d2, its ASCII code is smaller than that of the initial file path of thread A, so thread A is given priority to acquire the lock and perform the locking process.

[0132] In the above embodiments, by comparing the path values ​​of file paths corresponding to different file operation requests, the locking priority is determined based on the path value, thereby avoiding deadlock and improving locking efficiency.

[0133] This application also provides an application scenario, namely, the above file operation method is applied to the Hadoop Namenode metadata management node, as detailed below:

[0134] In native Hadoop, the FSDirectory (i.e., the innode tree) uses a single lock (dirLock) to protect access to and modification of the entire innode tree. Because the entire innode tree corresponds to a single lock, reading and writing unrelated files requires competition for the same lock, resulting in mutual exclusion and limiting concurrency in Namenode metadata operations. Here, an innode represents a file or directory.

[0135] Therefore, this application provides a solution as follows:

[0136] (a) Allocate a lock for the Inode in the Inode tree that needs to be locked.

[0137] Each inode in the inode tree can possess a lock. When operating on different inodes, locks are acquired for each inode, ensuring mutual exclusion for read and write operations on the same inode while preventing read and write operations between different inodes from affecting each other, thereby improving the Namenode's throughput. As shown in Figure 5, when a client performs a write operation on the file fn1 in the path qq / dn / fn1, it acquires a read lock on the root directory qq and a read lock on the dn directory, while acquiring a write lock on fn1. Other directories such as d11 and d1n are not locked, meaning that read and write operations on d11 and d1n are unaffected.

[0138] To accommodate the different locking modes of various Hadoop operations, the overall locking modes are described in three categories: READ type operations, WRITE type operations, and WRITE_PARENT type operations. (See reference...) Figure 9 The details are as follows:

[0139] (1) For READ type operations, read locks can be acquired from top to bottom according to the path. Among them, READ type operations mainly include: getFileInfo operation, getBlockLocation operation and listStatus operation.

[0140] (2) For WRITE type operations, since these operations only involve the modification of the last Inode, only the last Inode (i.e., the target file to be operated on) needs to be locked for writing, and the other Inodes need to be locked for reading. Among them, WRITE type operations include: setTime operation, setOwner operation and completeFile operation.

[0141] (3) For operations of type WRITE_PARENT, which can be file creation operations, mainly including create and mkdir operations. Since these operations not only modify the last Inode, but also the children in the parent directory of the Inode, write locks need to be added to the last Inode and its parent directory.

[0142] Besides different locking types, the locking order for directories needs to be consistent. Here, a top-down locking method is used to avoid deadlocks caused by circular waiting. Figure 10 As shown, there are three possible scenarios, as detailed below:

[0143] 1) Locking is performed sequentially from top to bottom according to the file path. Typical operations include: getFileInfo, delete, and mkdir.

[0144] 2) Locking is performed based on the file representation (fileId), mainly involving the allocateBlock and completeFile operations. These two operations do not provide a file path but rather a fileId. Therefore, the file path is obtained based on the fileId, and then locking is performed sequentially from top to bottom based on the obtained file path.

[0145] 3) Dual directory operations, which may include rename operations, require locking directories in both the source and destination file paths. This necessitates comparing the ASCII codes of the source and destination file paths, determining the order of locking based on the magnitude of the ASCII codes, and then locking in that order to avoid the risk of deadlock.

[0146] (ii) To avoid increasing the NameNode's memory overhead, a LockPool can be created, such as... Figures 6-8 As shown, when a lock is needed, a reference to the lock is obtained from the LockPool so that the directory and file can be locked based on the obtained lock; when the lock is no longer needed, it is released to reclaim the lock, as shown below. Figure 3 As shown, this saves memory overhead. For example, assuming a file operation request concurrency of 50, an inode tree depth of 100, and each request is a dual-directory operation, then a maximum of 50 × 100 × 2 = 10,000 lock objects (less than 1MB) would be needed. Compared to adding lock objects to all inodes, using LockPool saves a huge amount of memory.

[0147] like Figure 11As shown, when locking an Inode is required, it's not necessary to allocate a lock for each Inode. Instead, a lock is obtained from the LockPool. If the required lock exists in the LockPool, it can be acquired directly; otherwise, a reclaimed lock is obtained from the reuseLockPool, or a new lock is created. Furthermore, an asynchronous evictor can be created. This evictor reclaims unreferenced locks when the LockPool reaches a high watermark (i.e., the number of occupied locks reaches the upper threshold), continuing until it reaches a low watermark (i.e., the number of occupied locks is less than or equal to the lower threshold). References to reclaimed locks are placed in the reuseLockPool. When locking is needed, reclaimed locks are used preferentially, thus reducing memory garbage.

[0148] (iii) In order to deal with deadlock situations that may occur in the production environment, a lock view function has been introduced to facilitate timely location and detection of problems.

[0149] Add a lock timeout monitoring thread (monitorTimeoutLock) in the Namenode backend. Iterate through all locks in the LockPool. If a lock that is being held times out, the lock timeout monitoring thread will write the corresponding lock information (such as thread ID, holding time, and corresponding operation) to the log for problem analysis.

[0150] Through the solutions described in the above embodiments, under general big data processing scenarios (read to write ratio of 8:1) and with the same configuration, the improved Namenode performance of this application is 227% of the native Hadoop Namenode. Figure 12 As shown.

[0151] It should be understood that, although Figure 2 , 3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , 3 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0152] In one embodiment, such as Figure 13As shown, a file operation device is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: a determining module 1302, a searching module 1304, an operation module 1306, and an operation module 1308, wherein:

[0153] The determination module 1302 is used to determine the file path of the target file specified in the file operation request in response to the file operation request;

[0154] The search module 1304 is used to search for the file directory to which the target file belongs, level by level, from the start to the end of the file path.

[0155] The locking module 1306 is used to lock the file directories found level by level and the target file in turn during the process of searching for the file directory to which the target file belongs.

[0156] Operation module 1308 is used to perform file operations on the target file; during the write operation on the target file, other file operation requests for the target file are in a waiting state.

[0157] In the above embodiments, in response to a file operation request, the file path of the target file specified in the file operation request is determined; the file directory to which the target file belongs is searched level by level along the file path from the start point to the end point; during the process of searching the file directory to which the target file belongs, the file directories found level by level are locked sequentially, and the target file is also locked, thereby realizing the individual locking of the file directories and the target file corresponding to different nodes in the file path. This ensures that file operations on other different file paths are not affected during file operations on the target file, thus improving the data operation efficiency in the distributed computing platform and consequently increasing the throughput efficiency of the distributed computing platform. Furthermore, during the write operation on the target file, other file operation requests for the target file are in a waiting state, ensuring that the current write operation on the target file is not interrupted, which is beneficial to improving data accuracy.

[0158] In one embodiment, the determining module 1302 is further configured to read the file path of the target file from the file operation request; or, read the file identifier of the target file from the file operation request and find the file path of the target file based on the file identifier.

[0159] In one embodiment, the search module 1304 is further configured to obtain the directory node tree corresponding to the file operation request; the file path is a branch in the directory node tree that starts from the node corresponding to the root directory and ends at the node corresponding to the target file; in the directory node tree, the file directory is searched level by level along the file path starting from the node corresponding to the root directory to obtain the file directory including the target file at each level;

[0160] In the above embodiments, when a file operation request is received, only the directories and target files corresponding to the nodes in the file path within the directory node tree are locked, while the directories and files corresponding to the nodes in other branches are not locked. This ensures that the operations of other subdirectories under the same root directory in the directory node tree are not affected, avoiding the problem of reduced operation efficiency caused by locking the root directory of the entire target file in the distributed computing platform in the traditional solution, and effectively improving the efficiency of data operations in the distributed computing platform.

[0161] The locking module 1306 is also used to lock the file directories containing the target file found level by level in sequence.

[0162] In one embodiment, the file operation request includes a file read request;

[0163] The locking module 1306 is also used to obtain shared locks from the lock resource pool or create shared locks; lock the file directories found level by level based on different shared locks; and lock the target file according to different shared locks.

[0164] In one embodiment, the file operation request includes a file write request;

[0165] The locking module 1306 is also used to obtain shared locks and exclusion locks from the lock resource pool, or to create shared locks and exclusion locks; to lock the file directories found level by level based on different shared locks; and to lock the target file based on the exclusion lock.

[0166] In one embodiment, the file operation request includes a file creation request;

[0167] The locking module 1306 is also used to obtain shared locks and exclusive locks from the lock resource pool, or to create shared locks and exclusive locks; when the file directory found level by level is not the parent node of the target file, the found file directory is locked based on different shared locks; when the file directory found level by level is the parent node of the target file, the found file directory is locked based on exclusive locks; and the target file is locked based on different exclusive locks.

[0168] In the above embodiments, different types of locks and different locking strategies are used for different types of file operation requests. This ensures that while a write operation is being performed on the target file, other file operation requests for the target file are in a waiting state, guaranteeing that the current write operation is not interrupted and improving data accuracy. Furthermore, during a read operation on the target file, other file read requests can lock both the target file and its corresponding directory to ensure file sharing. Additionally, during the creation of a target file, since an exclusion lock is applied to the target file and its parent directory, other file operation requests targeting the target file and its parent directory are also in a waiting state, ensuring that the creation operation of the target file in the parent directory is not interrupted and that the target file is not affected by other requests, further improving data accuracy. Finally, each node in the directory node tree does not need to correspond to a single lock; instead, locks are acquired from a lock resource pool when needed, reducing the number of locks.

[0169] In one embodiment, the shared lock and the exclusive lock are different lock objects that include a reference counter; such as Figure 14 As shown, the device also includes:

[0170] The adjustment module 1310 is used to adjust the reference counters in the released shared lock and exclusive lock respectively when the shared lock and exclusive lock are released, so that the count in the adjusted reference counters can be used to indicate that the released shared lock and exclusive lock are not occupied.

[0171] The storage module 1312 is used to store the adjusted reference counter count, as well as the lock identifiers of shared locks and exclusive locks, in the lock recycling pool.

[0172] In one embodiment, such as Figure 14 As shown, the device also includes:

[0173] Generation module 1314 is used to generate a lock timeout monitoring thread;

[0174] The monitoring module 1316 is used to monitor the occupancy time of locks in the lock resource pool through the lock timeout monitoring thread;

[0175] The write module 1318 is used to write the lock information to the log through the lock timeout monitoring thread when the lock holding time reaches the time threshold.

[0176] Display module 1320 is used to display lock information in the log through the lock view when an abnormality occurs in a lock that is in an occupied state in the lock resource pool.

[0177] In one embodiment, the file operation request includes a first move request and a second move request to move the target file; the file path includes a first file path and a second file path corresponding to the first move request and the second move request, respectively;

[0178] The locking module 1306 is also used to lock the file directories found level by level in the first file path and lock the target file in turn when the path value of the first file path is less than the path value of the second file path, until the target file is operated on. Then, it locks the file directories found level by level in the second file path and locks the target file after the operation is completed.

[0179] In one embodiment, the locking module 1306 is further configured to, during the process of reading the target file, if a read operation request for other files of the target file is received, lock the file directories found level by level based on the read locks in the lock resource pool, and lock the target file in turn.

[0180] Operation module 1308 is also used to perform file reading operations on the target file.

[0181] In the above embodiments, by comparing the path values ​​of file paths corresponding to different file operation requests, the locking priority is determined based on the path value, thereby avoiding deadlock and improving locking efficiency.

[0182] For specific limitations regarding the file operation device, please refer to the limitations on file operation methods above, which will not be repeated here. Each module in the aforementioned file operation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0183] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as locks, file directories, and files. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a file operation method.

[0184] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0186] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A file operation method, characterized in that, The method includes: In response to a file operation request, determine the file path of the target file specified in the file operation request; The file path is searched level by level from the starting point to the ending point to find the file directory to which the target file belongs, and the target file itself is also located. During the process of searching the file directory to which the target file belongs, the file directories found level by level are locked according to the file operation type corresponding to the file operation request, and the target file is locked in turn. When the file operation request is a file write operation, the file operation is performed on the target file; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state, and other files in the first file path can continue to be operated on, the first file path containing at least one of the file directories; When the file operation request is a file creation operation, and the file creation operation includes create and mkdir operations, other file operation requests for the target file and file operations for the target file's parent directory are in a waiting state, and file operations can continue to be performed on files in other second file paths, which do not contain the parent directory but contain the target file's non-parent directory.

2. The method according to claim 1, characterized in that, Determining the file path of the target file specified in the file operation request includes: Read the file path of the target file from the file operation request; or... Read the file identifier of the target file from the file operation request, and find the file path of the target file based on the file identifier.

3. The method according to claim 1, characterized in that, The step-by-step search along the file path from the start to the end to find the file directory to which the target file belongs includes: Obtain the directory node tree corresponding to the file operation request; the file path is a branch in the directory node tree that starts from the node corresponding to the root directory and ends at the node corresponding to the target file; In the directory node tree, the file directory is searched level by level along the file path, starting from the node corresponding to the root directory, to obtain the file directory containing the target file at each level; The step of locking the file directories found level by level includes: The file directories containing the target file are locked sequentially as they are found level by level.

4. The method according to claim 1, characterized in that, The file operation request includes a file read request; the process of locking the file directories found level by level according to the file operation type corresponding to the file operation request, and locking the target file, includes: According to the file operation type corresponding to the file read request, obtain a shared lock from the lock resource pool, or create the shared lock; Based on the different shared locks, the file directories found level by level are locked sequentially; The target file is locked according to the different shared locks mentioned above.

5. The method according to claim 1, characterized in that, The file operation request includes a file write request; the process of locking the file directories found level by level according to the file operation type corresponding to the file operation request, and locking the target file, includes: According to the file operation type corresponding to the file write request, obtain a shared lock and an exclusive lock from the lock resource pool respectively, or create the shared lock and the exclusive lock; Based on the different shared locks, the file directories found level by level are locked sequentially; The target file is locked according to the exclusion lock.

6. The method according to claim 1, characterized in that, The file operation request includes a file creation request; the process of locking the file directories found level by level according to the file operation type corresponding to the file operation request, and locking the target file, includes: According to the file operation type corresponding to the file creation request, obtain a shared lock and an exclusive lock from the lock resource pool respectively, or create the shared lock and the exclusive lock; When the file directory found through the hierarchical search is not the parent node of the target file, the found file directory is locked based on different shared locks; When the file directory found through hierarchical searching is the parent node of the target file, the found file directory is locked based on the exclusion lock; The target file is locked based on different exclusion locks.

7. The method according to any one of claims 4 to 6, characterized in that, The shared lock and the exclusive lock are both different lock objects that include a reference counter; After performing file operations on the target file, the method further includes: When the shared lock and the exclusion lock are released, the reference counters in the released shared lock and the exclusion lock are adjusted so that the adjusted count in the reference counters indicates that the released shared lock and the exclusion lock are not occupied. The adjusted reference counter count, along with the lock identifiers of the shared lock and the exclusive lock, are stored in the lock recycling pool.

8. The method according to any one of claims 4 to 6, characterized in that, The method further includes: Thread for generating lock timeout monitoring; The lock timeout monitoring thread monitors the occupancy time of locks in the lock resource pool that are in an occupied state. When the lock's holding time reaches the time threshold, the lock information is written to the log by the lock timeout monitoring thread; When a lock in the lock resource pool that is in an occupied state becomes abnormal, the lock information in the log is displayed through the lock view.

9. The method according to claim 1, characterized in that, The file operation request includes a first move request and a second move request to move the target file; the file path includes a first file path and a second file path corresponding to the first move request and the second move request, respectively. The step of sequentially locking the file directories found level by level, and locking the target file, includes: When the path value of the first file path is less than the path value of the second file path, the file directories found level by level in the first file path are locked sequentially, and the target file is also locked, until the target file has been operated on completely. The file directories found level by level in the second file path are locked sequentially, and the target file after the operation is completed is also locked.

10. The method according to any one of claims 1 to 4, characterized in that, The method further includes: During the read operation on the target file, if a read operation request for another file on the target file is received, then Based on the read locks in the lock resource pool, the file directories found level by level are locked sequentially, and the target file is locked as well. Perform a file read operation on the target file.

11. A file operation device, characterized in that, The device includes: The determination module is used to determine the file path of the target file specified in the file operation request in response to the file operation request; The search module is used to search for the file directory to which the target file belongs, level by level, along the file path from the start point to the end point, and to find the target file; The locking module is used to lock the file directories found level by level according to the file operation type corresponding to the file operation request, and to lock the target file during the process of searching the file directory to which the target file belongs. An operation module is configured to perform file operations on the target file when the file operation request is a file write operation; wherein, during the write operation on the target file, other file operation requests for the target file are in a waiting state, and files in other first file paths can continue to be operated on, the first file paths containing at least one of the file directories; when the file operation request is a file creation operation, and the file creation operation includes create and mkdir operations, other file operation requests for the target file and file operations of the target file's parent directory are in a waiting state, and files in other second file paths can continue to be operated on, the second file paths not containing the parent directory but containing the non-parent directory of the target file.

12. The apparatus according to claim 11, characterized in that, The determining module is further configured to read the file path of the target file from the file operation request; or, read the file identifier of the target file from the file operation request and find the file path of the target file based on the file identifier.

13. The apparatus according to claim 11, characterized in that, The search module is further configured to obtain the directory node tree corresponding to the file operation request; the file path is a branch in the directory node tree that starts from the node corresponding to the root directory and ends at the node corresponding to the target file; In the directory node tree, the file directory is searched level by level along the file path, starting from the node corresponding to the root directory, to obtain the file directory containing the target file at each level; The locking module is also used to lock the file directories containing the target file found level by level in sequence.

14. The apparatus according to claim 11, characterized in that, The file operation request includes a file read request; The locking module is also used to obtain shared locks from the lock resource pool according to the file operation type corresponding to the file read request, or to create the shared locks. Based on different shared locks, the file directories found level by level are locked sequentially; based on different shared locks, the target file is locked.

15. The apparatus according to claim 11, characterized in that, The file operation request includes a file write request; The locking module is also used to obtain a shared lock and an exclusion lock from the lock resource pool according to the file operation type corresponding to the file write request, or to create the shared lock and the exclusion lock. Based on the different shared locks, the file directories found level by level are locked sequentially; The target file is locked according to the exclusion lock.

16. The apparatus according to claim 11, characterized in that, The file operation request includes a file creation request; The locking module is also used to obtain a shared lock and an exclusion lock from the lock resource pool according to the file operation type corresponding to the file creation request, or to create the shared lock and the exclusion lock. When the file directory found through the hierarchical search is not the parent node of the target file, the found file directory is locked based on different shared locks; When the file directory found through hierarchical searching is the parent node of the target file, the found file directory is locked based on the exclusion lock; The target file is locked based on different exclusion locks.

17. The apparatus according to any one of claims 14 to 16, characterized in that, The shared lock and the exclusive lock are both different lock objects that include a reference counter; The device further includes: An adjustment module is used to adjust the reference counters in the released shared lock and the released exclusive lock respectively when the shared lock and the exclusive lock are released, so that the count in the adjusted reference counter is used to indicate that the released shared lock and the exclusive lock are not occupied; The storage module is used to store the adjusted reference counter count, as well as the lock identifiers of the shared lock and the exclusion lock, in the lock recycling pool.

18. The apparatus according to any one of claims 14 to 16, characterized in that, The device further includes: The generation module is used to generate lock timeout monitoring threads; The monitoring module is used to monitor the occupancy time of locks in the lock resource pool through the lock timeout monitoring thread; The writing module is used to write the lock information of the lock into the log through the lock timeout monitoring thread when the lock occupancy time reaches the time threshold. The display module is used to display the lock information in the log through the lock view when an abnormality occurs in the lock resource pool that is in an occupied state.

19. The apparatus according to claim 11, characterized in that, The file operation request includes a first move request and a second move request to move the target file; the file path includes a first file path and a second file path corresponding to the first move request and the second move request, respectively. The step of sequentially locking the file directories found level by level, and locking the target file, includes: When the path value of the first file path is less than the path value of the second file path, the file directories found level by level in the first file path are locked sequentially, and the target file is also locked, until the target file has been operated on completely. The file directories found level by level in the second file path are locked sequentially, and the target file after the operation is completed is also locked.

20. The apparatus according to any one of claims 11 to 14, characterized in that, The device further includes: During the read operation on the target file, if a read operation request for another file on the target file is received, then Based on the read locks in the lock resource pool, the file directories found level by level are locked sequentially, and the target file is locked as well. Perform a file read operation on the target file.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

22. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images