File processing method and device, equipment, readable storage medium and program product

Abstract

The invention relates to a file processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a file operation request, wherein the file operation request comprises an operation path; sending the operation path to a metadata server, and receiving node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree; and processing a file under the target node corresponding to the file operation request according to the node information of the target node. By adopting the method, the file operation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a file processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of big data technology, enterprises and organizations are facing increasingly complex data storage and management challenges. As a core component of the big data ecosystem, Hadoop Distributed File System (HDFS) plays a key role in data storage and processing. However, with the expansion of business scale and the surge in data volume, a single HDFS cluster is often unable to meet all management needs, resulting in the increasing coexistence of multiple HDFS clusters.

[0003] In the related art, an independent distributed file system is used as a unified access layer for multiple file storage systems to achieve data access and management across storage systems. However, this cluster management method requires synchronizing the metadata of multiple storage clusters to the distributed file system, which increases the storage pressure of the distributed file system and results in low processing efficiency of the managed files. Summary of the invention

[0004] Based on this, it is necessary to provide a file processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the file processing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a file processing method, comprising:

[0006] Receiving a file operation request, wherein the file operation request includes an operation path;

[0007] Sending the operation path to a metadata server, and receiving node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree;

[0008] According to the node information of the target node, the file under the target node is processed corresponding to the file operation request.

[0009] In one embodiment, the processing corresponding to the file operation request on the file under the target node according to the node information of the target node includes:

[0010] According to the node information of the target node, access the child node of the target node, and obtain the node information of the child node from the metadata server;

[0011] According to the node information of the child node, traverse the nodes in the file cluster tree in sequence until the first node corresponding to the target file is determined, and obtain the node information of the first node from the metadata server;

[0012] The target file is acquired according to the node information of the first node, and a process corresponding to the file operation request is performed on the target file.

[0013] In one embodiment, acquiring the target file according to the node information of the first node, and processing the target file corresponding to the file operation request includes:

[0014] Determining a node type of the first node according to the node information of the first node;

[0015] If the node type of the first node is a symbolic link type, obtaining the target file through a proxy file system pointed to by the symbolic link corresponding to the first node;

[0016] If the node type of the first node is not a symbolic link type, obtaining the target file through the local file system;

[0017] The target file is processed in accordance with the file operation request.

[0018] In one embodiment, the file operation request further includes a target path; the method further includes:

[0019] Copying file data of a target file corresponding to the first node;

[0020] Obtaining node information of a second node corresponding to the target path sent by the metadata server;

[0021] According to the node information of the second node, the file data of the copied target file is saved in the target path corresponding to the second node.

[0022] In one embodiment, the file operation request includes a node rename request, and the node rename request includes a new name of the target node; the method further includes:

[0023] Creating a renamed node corresponding to the new name according to the operation path, and modifying the node identifier of the target node to the node identifier of the renamed node in the node information of the target node to obtain the node information of the renamed node;

[0024] A deletion instruction is sent to the metadata server to instruct the metadata server to delete the node information of the target node, and the node information of the renamed node is sent to the metadata server for storage.

[0025] In one embodiment, the method further comprises:

[0026] Receive a file mount request, wherein the file mount request includes a file mount path;

[0027] Create a corresponding mount node under the file mount path;

[0028] The node information of the mounting node is obtained, and the node information of the mounting node is sent to a metadata server for storage.

[0029] In a second aspect, the present application also provides a file processing device, comprising:

[0030] A request receiving module, used for receiving a file operation request, wherein the file operation request includes an operation path;

[0031] A metadata receiving module, used to send the operation path to a metadata server, and receive node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree;

[0032] The file processing module is used to process the file under the target node corresponding to the file operation request according to the node information of the target node.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the file processing method provided in the first aspect when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the file processing method provided in the first aspect are implemented.

[0035] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the file processing method provided in the first aspect.

[0036] The above-mentioned file processing method, device, computer equipment, computer-readable storage medium and computer program product receive a file operation request, send the operation path in the file operation request to the metadata server, and receive the node information of the target node corresponding to the operation path returned by the metadata server. According to the node information of the target node, the file under the target node is processed corresponding to the file operation request. It can be achieved that the node information of the cluster tree stored in the metadata server can be obtained on the client, and the file under the operation path can be conveniently obtained for processing according to the node information, thereby improving the file processing efficiency; because the metadata of the file cluster is stored in the metadata server, it is not limited to the number of stored clusters, and is suitable for large-scale file cluster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 An application environment diagram of a file processing method in an embodiment;

[0039] Figure 2 A schematic diagram of a flow chart of a file processing method in one embodiment;

[0040] Figure 3 A schematic diagram of a process for obtaining a target file in one embodiment;

[0041] Figure 4 A schematic diagram of the structure of a file cluster tree in one embodiment;

[0042] Figure 5 is a structural block diagram of a file processing device in one embodiment;

[0043] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The file processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the client 102 communicates with the metadata server 104 through the network, and the data storage system can store the data that the metadata server 104 needs to process. The data storage system can be integrated on the metadata server 104, or it can be placed on the cloud or other network servers. The metadata server 104 is used to store and manage the metadata corresponding to the HDFS file system 106, and the HDFS file system 106 can include multiple. The client 102 receives the file operation request and sends the operation path in the file operation request to the metadata server 104. The metadata server 104 sends the node information of the target node corresponding to the operation path to the client 102. The client 102 processes the file under the target node according to the node information of the target node corresponding to the file operation request. Among them, the file system 106 can be stored in the metadata server 104 according to the tree structure. The client 102 can be installed on the terminal, and the terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The metadata server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In an exemplary embodiment, Figure 2 As shown, a file processing method is provided, which is applied to Figure 1 The client in the example is used to illustrate, including the following steps 202 to 206. Among them:

[0047] Step 202: Receive a file operation request, where the file operation request includes an operation path.

[0048] The file operation request refers to a request initiated by a user to operate a file in the file system. For example, the file operation request may be initiated by an operation command, triggering an operation control, or entering a corresponding page.

[0049] Exemplarily, the file operation request may be, for example, a file access request, a file deletion request, a file mount request, a file copy request, a file rename request, etc. File operation requests for different application scenarios may include different contents, but the file operation request at least includes an operation path.

[0050] The operation path is used to characterize the storage location of the file in the file system. The formats of the operation paths corresponding to different operating systems may be different. For example, in the Linux operating system, the format of the operation path is / home / username / myfile.txt, and in the Windows operating system, the format of the operation path is C:\Users\username\Desktop, etc. It should be noted that in this embodiment, multiple file systems are stored in a tree structure through a unified naming method to obtain a file cluster tree. The entrance of the file cluster tree can be used as the root node, and each file system can be used as a child node of the root node. Then, according to the file structure of each file system, child nodes are generated downward in sequence until the leaf node corresponding to the final file. The operation path is a set of nodes starting from the root node in the file cluster tree to the branch where the target node corresponding to the target file is located.

[0051] Step 204: Send the operation path to the metadata server, and receive node information of the target node corresponding to the operation path returned by the metadata server; wherein the target node is a node in the file cluster tree.

[0052] The metadata server stores the node information of all nodes corresponding to the file cluster tree. The target node can be any node in the file cluster tree. Node information is information used to characterize the location and node attributes in the file cluster tree. Node information includes, for example, node directory information, node type, node access time, node modification time, and permissions. If the node type is a symbolic link type, the node information also includes the file system pointed to by the symbolic link. Node directory information is used to characterize the relationship between a node and the upper-level node.

[0053] The node types of nodes in the file cluster tree can include file type, path type and symbolic link type. The file type is used to indicate that the corresponding node corresponds to a file, and the path type indicates that the corresponding node corresponds to a path. The path type node is usually the middle node of the file cluster tree, and the symbolic link type indicates that the corresponding node corresponds to a symbolic link. A symbolic link (soft link) is a special type of file that contains a reference to other files or directories in the form of an absolute path or a relative path. A symbolic link can act as a shortcut. Through a symbolic link, the corresponding target file or directory can be accessed without knowing its location. It is easy to understand that a symbolic link can point to a file in the client local system or a file in the proxy file system.

[0054] It can be understood that each operation path has a corresponding node in the file cluster tree. After receiving the operation path, the metadata server can start from the root node and find the target node corresponding to the operation path in the file cluster tree according to the order of each node in the operation path, and then return the saved node information to the client in the form of a view. The view may include a tree structure or a table.

[0055] Step 206: Process the file under the target node according to the node information of the target node in accordance with the file operation request.

[0056] The processing corresponding to the file operation request may include, for example, access, copy, delete or rename processing. After obtaining the node information of the target node, the file under the target file node may be processed corresponding to the file operation request according to the node information. Exemplarily, the file operation request is a file deletion request. After obtaining the node information of the target node, a deletion instruction is sent to the metadata server to instruct the metadata server to delete the stored node information of the target node, and the file corresponding to the target node will also be deleted.

[0057] Optionally, the node type of the target node may be determined according to the node information of the target node, and the acquisition cluster of the files under the target node may be determined according to the node type of the target node.

[0058] The above-mentioned file processing method receives a file operation request, sends the operation path in the file operation request to the metadata server, and receives the node information of the target node corresponding to the operation path returned by the metadata server, and processes the file under the target node corresponding to the file operation request according to the node information of the target node. Since the file system is uniformly stored in the file cluster tree in the form of a tree structure, the metadata of the file system corresponding to the file cluster tree is stored in the metadata server, and the number and scale of the stored file systems are not limited. At the same time, the client can obtain the tree structure information of the file system metadata, which is convenient for processing files according to the tree structure information, and can improve the file processing efficiency.

[0059] In some embodiments, step 206 of performing processing corresponding to the file operation request on the file under the target node according to the node information of the target node includes:

[0060] According to the node information of the target node, the child nodes of the target node are accessed, and the node information of the child nodes is obtained from the metadata server; according to the node information of the child nodes, the nodes in the file cluster tree are traversed in sequence until the first node corresponding to the target file is determined, and the node information of the first node is obtained from the metadata server; according to the node information of the first node, the target file is obtained, and the target file is processed corresponding to the file operation request.

[0061] In an actual application scenario, the node information may also include the name of the child node; after obtaining the node information of the target node, the client can display the names of all the child nodes of the target node. When the user selects the name of one of the child nodes, in response to the selection trigger of the child node, the corresponding child node is accessed, and the operation path corresponding to the child node is sent to the metadata server. The metadata server returns the node information of the child node to the client according to the operation path corresponding to the child node, and then the client will display the name of the next level node of the child node, and so on, traversing the nodes in the file cluster tree in sequence until the operation path corresponding to the target file is determined, and the operation path of the target file is sent to the metadata server. The metadata server returns the node information of the first node corresponding to the target file, obtains the target file from the file system according to the node information of the first node, and processes the target file corresponding to the operation request, such as access, deletion, copying, renaming, etc. It is easy to understand that the file system involved in the embodiments of the present application refers to a distributed file system.

[0062] In one example, starting from the root node of the file cluster tree, traverse layer by layer until the first node corresponding to the target file is determined, and the operation path where the target file is located is sent to the metadata server. The metadata server returns the node information of the first node corresponding to the operation path where the target file is located, such as the node identifier of the first node, the node directory information of the first node, the node type of the first node, etc. The target file can be obtained according to the node type of the first node, and then the target file can be processed corresponding to the file operation request.

[0063] In this embodiment, based on the node information of the target node, the child nodes are accessed layer by layer starting from the target node in the file cluster tree until the first node corresponding to the target file is determined, the node information of the first node is obtained, and the target file is obtained based on the node information of the first node. Then, the target file is processed corresponding to the file operation request. This allows the target file to be operated to be determined by traversing the tree structure, and the target file can be found quickly, thereby improving the access efficiency of the target file.

[0064] In an exemplary embodiment, acquiring a target file according to the node information of the first node, and performing processing on the target file corresponding to the file operation request includes:

[0065] Determine the node type of the first node based on the node information of the first node; if the node type of the first node is a symbolic link type, obtain the target file through the proxy file system pointed to by the symbolic link corresponding to the first node; if the node type of the first node is not a symbolic link type, obtain the target file through the local file system; and perform processing on the target file corresponding to the file operation request.

[0066] The node information may include a node type, which is determined according to the node attributes when the node is created. The metadata server stores the correspondence between the node identifier and the node type in the file cluster tree, and the node type of the first node can be determined according to the correspondence between the node identifier and the node type. It can be understood that there is a one-to-one mapping relationship between the operation path, the node, and the node identifier. When the operation path is known, the corresponding node can be located and the corresponding node identifier can be obtained.

[0067] Optionally, if the node type of the first node is a symbolic link type, the node information also includes the corresponding bound file system information, and the metadata server stores the correspondence between the node identifier and the bound file system information. Based on the correspondence between the node identifier and the bound file system information, the file system corresponding to the first node can be determined, and then the target file can be obtained from the corresponding file system.

[0068] In one example, if Figure 3 As shown, starting from the root node, traverse layer by layer until the target file is determined. The target file is usually a file that the user needs to operate. According to the operation path where the target file is located, the node identifier of the first node is determined from the directory information table, and the node type of the first node is obtained from the node attribute table. If the node type of the first node belongs to the symbolic link type, the file system address bound to the first node is obtained from the node path table, and it is identified whether the bound file system address starts with hdfs: / / . If so, the bound file system address, that is, the proxy file system path, is used to initialize the proxy file system. The proxy file system can be a remote file system, and then the target file is obtained from the proxy file system; if the bound file system address does not start with hdfs: / / , the target file is obtained through the local file system. If the node type of the first node does not belong to a symbolic link, the target file is obtained from the local file system of the client. Among them, the directory information table is used to store information between the node and the upper-level node. For example, the directory information table stores key-value pairs, and the key of each key-value pair is the node identifier of the upper-level node and the node name of the current node, and the value is the node identifier of the current node. The node attribute table is used to store the correspondence between the node identifier and the node type, and the node path table is used to store the correspondence between the node identifier and the bound file system address.

[0069] In this embodiment, the target file is obtained according to the node type of the first node. If the node type of the first node is a symbolic link type, the target file is obtained through the proxy file system pointed to by the symbolic link corresponding to the first node. If the node type of the first node is not a symbolic link type, the target file is obtained through the local file system, and then the target file is processed corresponding to the file operation request. The remote proxy file system can be added to the file cluster tree for management through the symbolic link, which makes the file system management more lightweight and convenient, and further improves the file access efficiency.

[0070] In some embodiments, the file operation request further includes a target path; the above method further includes:

[0071] Copy the file data of the target file corresponding to the first node; obtain the node information of the second node corresponding to the target path sent by the metadata server; and save the file data of the copied target file in the target path corresponding to the second node according to the node information of the second node.

[0072] The target path refers to the path where the file is saved. The file operation request includes a file copy request, and the file copy request includes the path where the copied file is saved.

[0073] In an actual application scenario, after the client obtains the target file, it copies the file data of the target file, and then, according to the target path, starts from the root node of the file cluster tree, traverses layer by layer until it reaches the second node corresponding to the target path, obtains the node information of the second node sent by the metadata server, and saves the file data of the copied target file under the target path corresponding to the second node according to the node information of the second node. It is easy to understand that the process of obtaining the node information of the second node corresponding to the target path sent by the metadata server is similar to the process of obtaining the node information of the target node corresponding to the operation path returned by the metadata server.

[0074] Exemplarily, a child node may be created under the second node, and the file data of the copied target file may be stored under the child node. It is easy to understand that the node type of the created child node may be a file type, for example, the created child node corresponds to a specific folder, and the file data of the copied target file is stored in the folder.

[0075] In this embodiment, file copying is achieved by copying the file data of the target file corresponding to the first node, and then saving the copied file data under the target path according to the node information of the second node corresponding to the target path. Since the node information of the second node can be quickly obtained according to the node information, the copied file data can be quickly saved under the target path corresponding to the second node, thereby achieving rapid copying and saving of files, that is, improving file operation efficiency.

[0076] In some embodiments, the file operation request includes a node rename request, and the node rename request includes a new name of the target node; the above method further includes:

[0077] A renamed node corresponding to the new name is created according to the operation path, and in the node information of the target node, the node identifier of the target node is modified to the node identifier of the renamed node to obtain the node information of the renamed node; a delete instruction is sent to the metadata server to instruct the metadata server to delete the node information of the target node, and the node information of the renamed node is sent to the metadata server for storage.

[0078] The node renaming request refers to a request for renaming a node. The node renaming request includes a new name of the target node, and the new name is the name of the node after the name is renamed.

[0079] During the node renaming operation, the node name of the target node corresponding to the operation path can be called the old name. In the actual application scenario, when the client receives the renaming request, it recreates the rename node corresponding to the new name according to the operation path, and assigns the corresponding node identifier of the rename node, and then changes the node identifier of the target node to the node identifier of the rename node in the node information of the target node, obtains the node information of the rename node, sends the node information of the rename node to the metadata server for storage, and sends a delete command to the metadata server to instruct the metadata server to delete the node information of the target node.

[0080] Optionally, the node information of the target node includes the target node identifier, a key-value pair consisting of the target node identifier and the upper node information, a node type, etc. If the node type of the target node is a symbolic link type, the node information of the target node also includes the target node identifier and the corresponding bound file system information. Then, in the target node information, the target node identifier is modified to a renamed node identifier, and the node information of the renamed node is obtained, including a key-value pair consisting of the renamed node identifier and the upper node information, the node type of the renamed node, and the renamed node and the corresponding bound file system information.

[0081] In this embodiment, a rename node corresponding to the new name is created according to the operation path, and the node identifier of the target node is modified to the node identifier of the rename node in the node information of the target node to obtain the node information of the rename node. The node information of the target node is deleted through the metadata server, and the node information of the rename node is saved to realize node renaming. The file path can be flexibly managed, and the organization and classification of files are convenient. Users can transparently rename files based on the file cluster tree without worrying about the actual storage location of the files, thereby improving file operation efficiency.

[0082] In some embodiments, the above method further comprises:

[0083] A file mount request is received, wherein the file mount request includes a file mount path; a corresponding mount node is created under the file mount path; node information of the mount node is obtained, and the node information of the mount node is sent to a metadata server for storage.

[0084] The file mount request may be represented in the form of a file mount command, and the file mount request includes a file mount path, that is, a location where the file is mounted in the file cluster tree.

[0085] Optionally, according to the file mount path, the corresponding position is found in the file cluster tree, a mount node is created, and a node identifier is assigned to the mount node. The node identifier of the upper node of the mount node and the node name of the mount node are used as the key, and the node identifier of the mount node is used as the value to form a key-value pair, the key-value pair is used as the node directory information of the mount node, the node directory information, node type and bound file system information of the mount node are used as the node information of the mount node, and the node information of the mount node is sent to the metadata server for storage, so as to facilitate subsequent access processing of the mount node.

[0086] In this embodiment, by creating a corresponding mount node under the file mount path, the node information of the mount node is sent to the metadata for storage, that is, the mount node is added to the file cluster tree and managed as a node in the file cluster tree, which facilitates flexible and orderly management of files, thereby improving file access and processing efficiency.

[0087] In an exemplary embodiment, the client can integrate a multi-view management SDK (Software Development Kit), and the HDFS file system can be a remote file system and / or a local file system, and the HDFS file system can include multiple. The metadata of the HDFS file system can be stored in a metadata server, and the metadata server can be stored through storage methods such as mysql, postgresql, and redis. In actual application scenarios, if the application service needs to read and write a distributed file system, an operation request can be initiated to the metadata server. For example, an HDFS shell command such as hdfsdfs-get or hdfs dfs-put can be used for operation. These operation requests will pass through the virtual file system layer of the client SDK. The virtual file system layer inherits and implements all interface semantics of the HDFS FileSystem. FileSystem is a core class that can provide a rich API (Application Programming Interface) for operating files and directories on the HDFS file system. The objects include node objects (inode) and directory objects (entry). The node object records the metadata information of the file system, and the directory object records the structural information of the file system, that is, the hierarchical relationship between different directories. The information of the node object and the target object is stored in the metadata server. The metadata server manages the file system hierarchy, file attributes, permissions, and other information in the unified namespace.

[0088] For example, the file cluster tree structure is as follows: Figure 4 As shown, Figure 4The file cluster tree in includes two HDFS file clusters, where file cluster hdfs nn1 is bound to the / data path, and file cluster hdfs nn2 is bound to the / user path. It is easy to understand that the number of file clusters that can be included in the file cluster tree in the actual application scenario is not limited to two, but can be hundreds, thousands or more. The number of file clusters that can be included in the file cluster tree is not limited. The URI (Uniform Resource Identifier) ​​entry of the file cluster tree is taken as the root node, and the node identifier of the root node is represented by 0. Then, the path is represented by the node names of each level in order from top to bottom. For example, the node name corresponding to the first child node of the root node is data, and the node identifier can be represented by 1. Then the path corresponding to the first child node is / data, and the node name of the root node can be omitted. Similarly, the node name of the second child node of the root node is user, and the node identifier can be represented by 2, and the corresponding path is / user. Because the node type of the node corresponding to / data is a symbolic link type and the corresponding bound file cluster is hdfs nn1, an operation request can be initiated to the metadata server so that the metadata server returns the metadata information corresponding to the file cluster hdfs nn1. Therefore, the client can synchronously display the metadata information of the file cluster hdfs nn1.

[0089] For example, in Figure 4 In the file cluster tree shown, the corresponding node directory information can be shown in Table 1 below. The node directory information can be represented in the form of a key-value pair, where the key is composed of the node identifier of the parent node of the current node and the node name of the current node, and the value is the node identifier of the current node. For example, the parent node corresponding to the first child node is the root node, the node identifier of the root node is 0, and the node name of the current node is data, then the key is 0 / data, and the value is the node identifier 1 of the first child node.

[0090] Table 1

[0091]

[0092] For example, Figure 4 The node attribute information corresponding to the file cluster tree shown is shown in Table 2 below. Table 2 only shows the attributes of the node type. The node attribute information can also include attributes such as file access time, file modification time, and file access permission. The file cluster information bound to the node whose node type is a symbolic link type is shown in Table 3.

[0093] Table 2

[0094]

[0095] Table 3

[0096]

[0097] Exemplarily, the client obtains a file mount request corresponding to the mount command mount hdfs: / / nn2 / directory / user, creates a / user node under the mount path in the mount command, assigns node ID 2 to the node, adds node ID 2 and the corresponding node type "symbolic link" to the node attribute information in Table 2, adds the file cluster hdfs: / / nn2 / directory bound to the node ID 2 and the symbolic link to Table 3, and then adds "0 / user" and node ID 2 to Table 1.

[0098] For example, the process of file access is explained by querying / user / bob. According to the query path, start from the root node and traverse layer by layer until the node corresponding to the bob file is determined. From Table 1, it can be seen that the corresponding node identifier is 2. From the node attribute information, it can be obtained that the node type of node 2 is a symbolic link. From Table 3, the file system path pointed to by the symbolic link is hdfs: / / nn2 / directory. The proxy file system is used to open and access hdfs: / / nn2 / directory / bob.

[0099] For example, take the process of copying / data / reports to the / user directory as an example to explain the file copying process. First, access the / data / reports file, and finally access the path hdfs: / / nn1 / directory / reports to read the contents of the file. Then open the / user directory, and finally open the hdfs: / / nn2 / directory / directory, create the reports file in this directory, and copy the contents of the file read in the previous step to the hdfs: / / nn2 / directory / directory.

[0100] Exemplarily, if the path corresponding to the renamed node is before the node whose node type is a symbolic link type, the renaming is implemented by the local file system. If the path corresponding to the renamed node is after the node whose node type is a symbolic link type, the renaming is performed by the remote file system. Taking the renaming of / data to / store as an example, the process of renaming implemented by the local file system is described. According to the / data path, a / store node is created in the file cluster tree, and node ID 3 is assigned. Node ID 3 and the file type symbolic link are added to Table 2, node ID 3 and hdfs: / / nn1 / directory are added to Table 3, and then "0 / stpre" points to node ID 3. And the information related to node ID 2 is deleted in Table 1, Table 2 and Table 3.

[0101] For example, when deleting a file, start from the root node, traverse layer by layer, determine the node corresponding to the file to be deleted, and determine the node type of the node. If the node type is a symbolic link type, the remote file system agent deletes the lower-level files or directories, and the deleted files or directories will enter the recycling directory of the remote file cluster and be physically deleted by the remote file cluster at a fixed time. If the node type is not a symbolic link, continue to the next node. If the node type of the next node is not a symbolic link type, the local system deletes the corresponding file.

[0102] In the above embodiment, the file system is managed through multiple views, and multiple file systems are uniformly named and managed according to a tree structure. The view mapping and file access can be decoupled from the traffic, and transparent access to files distributed in different file systems can be achieved on the client, making the file system management more flexible and lightweight, improving the file access efficiency, and thus improving the file processing efficiency.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a file processing device for implementing the file processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more file processing device embodiments provided below can refer to the limitations on the file processing method above, and will not be repeated here.

[0105] In an exemplary embodiment, Figure 5 As shown, a file processing device is provided, including: a request receiving module 502, a metadata receiving module 504 and a file processing module 506, wherein:

[0106] The request receiving module 502 is used to receive a file operation request, wherein the file operation request includes an operation path;

[0107] The metadata receiving module 504 is used to send the operation path to the metadata server and receive the node information of the target node corresponding to the operation path returned by the metadata server; the target node is a node in the file cluster tree;

[0108] The file processing module 506 is used to process the file under the target node corresponding to the file operation request according to the node information of the target node.

[0109] In some embodiments, the file processing module 506 is also used to access the child nodes of the target node according to the node information of the target node, and obtain the node information of the child nodes from the metadata server; according to the node information of the child nodes, traverse the nodes in the file cluster tree in sequence until the first node corresponding to the target file is determined, and obtain the node information of the first node from the metadata server; according to the node information of the first node, obtain the target file, and process the target file corresponding to the file operation request.

[0110] In some embodiments, the file processing module 506 is also used to determine the node type of the first node based on the node information of the first node; if the node type of the first node is a symbolic link type, the target file is obtained through the proxy file system pointed to by the symbolic link corresponding to the first node; if the node type of the first node is not a symbolic link type, the target file is obtained through the local file system; and the target file is processed corresponding to the file operation request.

[0111] In some embodiments, the file operation request also includes a target path; the file processing module 506 is also used to copy the file data of the target file corresponding to the first node; obtain the node information of the second node corresponding to the target path sent by the metadata server; and save the file data of the copied target file under the target path corresponding to the second node according to the node information of the second node.

[0112] In some embodiments, the file operation request includes a node rename request, and the node rename request includes the new name of the target node; the file processing device also includes a renaming module, which is used to create a rename node corresponding to the new name according to the operation path, and in the node information of the target node, modify the node identifier of the target node to the node identifier of the renamed node to obtain the node information of the renamed node; send a delete instruction to the metadata server to instruct the metadata server to delete the node information of the target node, and send the node information of the renamed node to the metadata server for storage.

[0113] In some embodiments, the file processing device also includes a file mounting module, which is used to receive a file mounting request, the file mounting request includes a file mounting path; create a corresponding mounting node under the file mounting path; obtain the node information of the mounting node, and send the node information of the mounting node to the metadata server for storage.

[0114] Each module in the above-mentioned file processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a file processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0116] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the file processing method in the above embodiment are implemented.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the file processing method in the above embodiment are implemented.

[0119] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the file processing method in the above embodiment when executed by a processor.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file processing method, characterized in that: The method comprises: Receiving a file operation request, wherein the file operation request includes an operation path; Sending the operation path to a metadata server, and receiving node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree; According to the node information of the target node, the file under the target node is processed corresponding to the file operation request.

2. The method according to claim 1, characterized in that The processing corresponding to the file operation request on the file under the target node according to the node information of the target node includes: According to the node information of the target node, access the child node of the target node, and obtain the node information of the child node from the metadata server; According to the node information of the child node, traverse the nodes in the file cluster tree in sequence until the first node corresponding to the target file is determined, and obtain the node information of the first node from the metadata server; The target file is acquired according to the node information of the first node, and a process corresponding to the file operation request is performed on the target file.

3. The method according to claim 2, characterized in that The acquiring the target file according to the node information of the first node, and performing processing on the target file corresponding to the file operation request, includes: Determining a node type of the first node according to the node information of the first node; If the node type of the first node is a symbolic link type, obtaining the target file through a proxy file system pointed to by the symbolic link corresponding to the first node; If the node type of the first node is not a symbolic link type, obtaining the target file through the local file system; The target file is processed in accordance with the file operation request.

4. The method according to claim 2, characterized in that: The file operation request also includes a target path; the method also includes: Copying file data of a target file corresponding to the first node; Obtaining node information of a second node corresponding to the target path sent by the metadata server; According to the node information of the second node, the file data of the copied target file is saved in the target path corresponding to the second node.

5. The method according to claim 1, characterized in that The file operation request includes a node rename request, and the node rename request includes a new name of the target node; the method further includes: Creating a renamed node corresponding to the new name according to the operation path, and modifying the node identifier of the target node to the node identifier of the renamed node in the node information of the target node to obtain the node information of the renamed node; A deletion instruction is sent to the metadata server to instruct the metadata server to delete the node information of the target node, and the node information of the renamed node is sent to the metadata server for storage.

6. The method according to claim 1, characterized in that The method further comprises: Receive a file mount request, wherein the file mount request includes a file mount path; Create a corresponding mount node under the file mount path; The node information of the mounting node is obtained, and the node information of the mounting node is sent to a metadata server for storage.

7. A file processing device, characterized in that: The device comprises: A request receiving module, used for receiving a file operation request, wherein the file operation request includes an operation path; A metadata receiving module, used to send the operation path to a metadata server, and receive node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree; The file processing module is used to process the file under the target node corresponding to the file operation request according to the node information of the target node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images