Data processing method and distributed file system

Abstract

The present application discloses a data processing method and a distributed file system. The method is applicable to a front-end device in a distributed file system, comprising: obtaining a first operation request for a first target file, the first operation request including: path information corresponding to the first target file; using a path index table, according to the longest match principle, matching a target directory corresponding to the path information; querying the path index table to see whether the sharding status of the target directory is shardable, and if so, querying the sharding type corresponding to the target directory; if the sharding type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type. The technical solutions provided in each embodiment of the present application can improve the efficiency of user access to directories or files.

Description

Technical Field

[0001] The present application belongs to the field of storage technology, and in particular relates to a data processing method and a distributed file system. Background Art

[0002] In the prior art, it is generally believed that there are two types of data in a file system, depending on the different stored content: file system metadata, which includes directory / file attributes, extended attributes, permissions, and the part used to manage file data; and file system data, which is the file data that is truly valid for users.

[0003] Among them, the file system metadata recursively constructs a file system directory tree through the hierarchical relationship between parent directories and child directories. In traditional stand-alone file systems, file system metadata and file system data are persistently stored on the same physical node and one or more disks. However, the storage space of a single physical node is usually not very large (tens of TB). Even if the file system metadata and file system data can be saved on the same physical node through extended interfaces, for reasons such as data redundancy, data reliability, and throughput performance, the metadata and data need to be saved on different physical nodes, and the metadata needs to be sharded.

[0004] The directory sharding scheme in the existing technology generally performs consistent hash calculation on the absolute path name of the file or directory, and hashes the file or directory to different storage nodes in the distributed system. When a user accesses the file or directory, the distributed system will access the storage nodes one by one according to the user's access request to find the corresponding directory. The search process is cumbersome and the efficiency of user access to files or directories is low. Summary of the Invention

[0005] The embodiments of the present application provide an implementation solution that is different from the prior art to improve the efficiency of user access to metadata.

[0006] Specifically, in one embodiment of the present application, a data processing method is provided, which is applicable to a front-end device in a distributed file system, including: obtaining a first operation request for a first target file, the first operation request including: path information corresponding to the first target file; using a path index table, according to a longest match principle, matching a target directory corresponding to the path information; inquiring in the path index table whether the sharding status of the target directory is shardable, and if so, inquiring about the sharding type corresponding to the target directory; if the sharding type is a first preset type, determining a first target metadata server for executing a task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type.

[0007] Optionally, the method further includes:

[0008] If the shard type is the second preset type, the second target metadata server is determined based on the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that executes the task corresponding to the first operation request.

[0009] Optionally, determining, according to a first sharding algorithm corresponding to the first preset type, a first target metadata server for executing the task corresponding to the first operation request includes:

[0010] Determine the subdirectory corresponding to the target directory from the path information;

[0011] Using the lower-level directory, calculate a first calculation result using the first sharding algorithm;

[0012] Determining an index path corresponding to the first target file according to the first calculation result and the target directory;

[0013] The first target metadata server corresponding to the index path is searched from the path index table.

[0014] Optionally, the path information includes a file name of the first target file. When the type of the first operation request is a create request, the second target metadata server uses the first operation request to determine a third target metadata server that executes a task corresponding to the first operation request, including:

[0015] According to the file name, a second calculation result is calculated using a second sharding algorithm corresponding to the second preset type, and the third target metadata server is determined according to the second calculation result.

[0016] Optionally, the method further includes:

[0017] Adjust the path information in the first operation request according to the first calculation result to obtain a second operation request;

[0018] The second operation request is sent to the first target metadata server, so that the first target metadata server performs the task corresponding to the first operation request based on the second operation request.

[0019] Optionally, before obtaining the first operation request for the first target file, the method further includes:

[0020] receiving management configuration information, wherein the management configuration information includes the shard type for the target directory;

[0021] The fragmentation type is set in the path index table based on the management configuration information.

[0022] Optionally, the method further includes:

[0023] receiving a virtual directory setting instruction for the target directory;

[0024] Setting multiple virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

[0025] The plurality of virtual directory information includes the index path.

[0026] Optionally, when the directory corresponding to the first target file is a top-level subdirectory of the target directory, and the third target metadata server has executed the task corresponding to the first operation request, the method further includes:

[0027] Obtaining update information for the path index table sent by the configuration device;

[0028] Updating the path index table using the update information;

[0029] The update information is obtained by the configuration device according to feedback information sent by the third target metadata server after executing the task corresponding to the first operation request.

[0030] Optionally, the first target metadata server includes multiple virtual metadata servers;

[0031] Searching the path index table for the first target metadata server corresponding to the index path includes:

[0032] Determining a target virtual metadata server corresponding to the index path according to the path index table;

[0033] The first target metadata server is determined based on the target virtual metadata server.

[0034] Optionally, the first preset type includes hash sharding or sharding according to time information; the second preset type includes random sharding.

[0035] Optionally, the type of the first operation request includes any one of the following: a create request, a delete request, a modify request, and a query request.

[0036] In another embodiment of the present application, a data processing method is provided, which is applicable to a first target metadata server in a distributed file system, including: receiving a first operation request sent by a front-end device, the first operation request including: path information corresponding to the first target file; executing a task corresponding to the first operation request; wherein, after the first operation request is obtained by the front-end device, the path index table is used to match the target directory corresponding to the path information according to the longest match principle, and when it is queried in the path index table that the sharding status of the target directory is shardable, and the sharding type corresponding to the target directory is queried as a first preset type, according to the first sharding algorithm corresponding to the first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined and forwarded to the first target metadata server.

[0037] In another embodiment of the present application, a data processing device is provided, which is suitable for a front-end device in a distributed file system, including: an acquisition module, used to obtain a first operation request for a first target file, wherein the first operation request includes: path information corresponding to the first target file; a matching module, used to use a path index table to match a target directory corresponding to the path information according to a longest match principle; a query module, used to query in the path index table whether the sharding status of the target directory is shardable, and if so, query the sharding type corresponding to the target directory; a determination module, used to determine, if the sharding type is a first preset type, according to a first sharding algorithm corresponding to the first preset type, a first target metadata server that executes the task corresponding to the first operation request.

[0038] Optionally, the above-mentioned determination module is also used to: if the shard type is a second preset type, determine the second target metadata server based on the target directory and the path index table, and send the first operation request to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that executes the task corresponding to the first operation request.

[0039] Optionally, when the determination module is used to determine the first target metadata server for executing the task corresponding to the first operation request according to the first sharding algorithm corresponding to the first preset type, it is specifically used to:

[0040] Determine the subdirectory corresponding to the target directory from the path information;

[0041] Using the lower-level directory, calculate a first calculation result using the first sharding algorithm;

[0042] Determining an index path corresponding to the first target file according to the first calculation result and the target directory;

[0043] The first target metadata server corresponding to the index path is searched from the path index table.

[0044] Optionally, the path information includes a file name of the first target file. When the type of the first operation request is a create request, the second target metadata server, when using the first operation request to determine a third target metadata server that executes a task corresponding to the first operation request, is specifically configured to:

[0045] According to the file name, a second calculation result is calculated using a second sharding algorithm corresponding to the second preset type, and the third target metadata server is determined according to the second calculation result.

[0046] Optionally, the device is further used for:

[0047] Adjust the path information in the first operation request according to the first calculation result to obtain a second operation request;

[0048] The second operation request is sent to the first target metadata server, so that the first target metadata server performs the task corresponding to the first operation request based on the second operation request.

[0049] Optionally, before the acquisition module acquires the first operation request for the first target file, the apparatus is further configured to:

[0050] receiving management configuration information, wherein the management configuration information includes the shard type for the target directory;

[0051] The fragmentation type is set in the path index table based on the management configuration information.

[0052] Optionally, the device is further used for:

[0053] receiving a virtual directory setting instruction for the target directory;

[0054] Setting multiple virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

[0055] The plurality of virtual directory information includes the index path.

[0056] Optionally, when the directory corresponding to the first target file is the highest-level subdirectory of the target directory, and the third target metadata server has executed the task corresponding to the first operation request, the apparatus is further configured to:

[0057] Obtaining update information for the path index table sent by the configuration device;

[0058] Updating the path index table using the update information;

[0059] The update information is obtained by the configuration device according to feedback information sent by the third target metadata server after executing the task corresponding to the first operation request.

[0060] Optionally, the first target metadata server includes multiple virtual metadata servers;

[0061] When the determining module is configured to search the path index table for the first target metadata server corresponding to the index path, the determining module is specifically configured to:

[0062] Determining a target virtual metadata server corresponding to the index path according to the path index table;

[0063] The first target metadata server is determined based on the target virtual metadata server.

[0064] Optionally, the first preset type includes hash sharding or sharding according to time information; the second preset type includes random sharding.

[0065] Optionally, the type of the first operation request includes any one of the following: a create request, a delete request, a modify request, and a query request.

[0066] In another embodiment of the present application, a data processing device is provided, which is applicable to a first target metadata server in a distributed file system, including: a receiving module, used to receive a first operation request sent by a front-end device, the first operation request including: path information corresponding to the first target file; an execution module, used to execute a task corresponding to the first operation request; wherein, after the first operation request is obtained by the front-end device, the path index table is used to match the target directory corresponding to the path information according to the longest match principle, and when it is queried in the path index table that the sharding status of the target directory is shardable and the sharding type corresponding to the target directory is queried as a first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined according to the first sharding algorithm corresponding to the first preset type, and then forwarded to the first target metadata server.

[0067] In another embodiment of the present application, a distributed file system is provided, comprising: a front-end device, a configuration device, and at least one server; wherein: the front-end device is used to obtain a first operation request for a first target file, the first operation request including: path information corresponding to the first target file; using a path index table, according to a longest match principle, matching a target directory corresponding to the path information; in the path index table, querying whether the sharding status of the target directory is shardable, and if so, querying the sharding type corresponding to the target directory; if the sharding type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type; the first target metadata server in the at least one server is used to execute the task corresponding to the first operation request; the configuration device is used to obtain configuration instructions from relevant personnel, set the path index table according to the configuration instructions, and send the path index table to the front-end device when obtaining an index table acquisition request from the front-end device.

[0068] The present application provides a new solution that is different from the existing technology. The front-end device in the distributed file system can determine the server that actually processes the task corresponding to the received first operation request based on the path index table and the path information corresponding to the first target file, combined with the shard type of the target directory corresponding to the determined path information. This improves the efficiency of determining the server that processes the task corresponding to the first operation request, and further improves the user's access efficiency to files or directories. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0070] Figure 1 A schematic diagram of the structure of a distributed file system provided in one embodiment of the present application;

[0071] Figure 2a A flowchart of a data processing method provided in one embodiment of the present application;

[0072] Figure 2b A schematic diagram of the structure of a directory tree in a distributed file system provided in one embodiment of the present application;

[0073] Figure 2c A schematic diagram of the structure of a directory tree in another distributed file system provided in one embodiment of the present application;

[0074] Figure 3 A flowchart of a data processing method provided in another embodiment of the present application;

[0075] Figure 4 A flowchart of a data processing method provided in one embodiment of the present application;

[0076] Figure 5 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0077] Figure 6 A structural diagram of a data processing device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0078] Before introducing the technical solutions provided by the embodiments of this application, a brief introduction to the technical terms involved in this article is first given.

[0079] MDS: Metadata Server, metadata server;

[0080] v-MDS: virtual-Metadata Server, virtual metadata server.

[0081] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the situation of including at least one. It should be understood that the descriptions of "first", "second" and the like herein are used to distinguish different elements, devices, etc., do not represent a sequential order, and do not limit "first" and "second" to be different types. Depending on the context, the words "if", "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining" or "in response to monitoring". Similarly, depending on the context, the phrases "if it is determined" or "if monitoring (statement condition or event)" can be interpreted as "when determining" or "in response to determining" or "when monitoring (statement condition or event)" or "in response to monitoring (statement condition or event)".

[0083] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0084] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0085] Figure 1 A schematic diagram of the structure of a distributed file system provided in an exemplary embodiment of the present application includes: a front-end device 11, a configuration device 12, and at least one server 13; wherein:

[0086] The front-end device 11 is configured to obtain a first operation request for a first target file, the first operation request including: path information corresponding to the first target file; using a path index table, according to a longest match principle, matching a target directory corresponding to the path information; querying the path index table to determine whether the sharding status of the target directory is shardable; if so, querying the sharding type corresponding to the target directory; if the sharding type is a first preset type, determining a first target metadata server for executing a task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type;

[0087] A first target metadata server in the at least one server 13 is configured to execute a task corresponding to the first operation request;

[0088] The configuration device 12 is configured to obtain configuration instructions from relevant personnel, set the path index table according to the configuration instructions, and transmit the path index table to the front-end device upon receiving an index table acquisition request from the front-end device. Furthermore, the configuration device 12 can also provide relevant personnel with services for adding, deleting, modifying, and querying one or more index items in the path index table.

[0089] Furthermore, if the shard type is the second preset type, the front-end device 11 is further configured to determine a second target metadata server based on the target directory and the path index table, and send the first operation request to the second target metadata server, so that the second target metadata server uses the first operation request to determine a third target metadata server that executes the task corresponding to the first operation request;

[0090] The second target metadata server in the at least one server 13 is configured to determine the third target metadata server using the first operation request when the shard type is the second preset type;

[0091] In some optional embodiments of the present application, the first operation request for the first target file obtained by the front-end device can be determined based on the third operation request triggered by the user's operation on the first target file. When the type of the third operation request is a deletion request, a modification request, and a query request, the first operation request can be the same as the third operation request; when the type of the third operation request is a creation request, the first operation request can be determined based on the file name of the first target file generated according to preset rules and the third operation request.

[0092] In other optional embodiments of the present application, when the first operation request is any one of a creation request, a deletion request, a modification request, and a query request, the front-end device can obtain the first operation request directly from the user device. When the type of the first operation request is a creation request, the user device can generate the file name of the first target file in advance according to preset rules, and generate the first operation request based on the file name of the first target file.

[0093] In some optional embodiments of the present application, the front-end device 11 can be a device connected to a user device (such as a computer, a mobile device) used by the user, and the number can be one or more. When the user adds files, deletes files, modifies files, queries files, etc. based on the user device, the user device sends the corresponding operation request to the front-end device 11, so that the front-end device 11 determines the server that performs the corresponding task based on the operation request received from the user device.

[0094] Furthermore, the above-mentioned first target metadata server can be a server in a distributed file system that is directly determined by the front-end device to execute the task corresponding to the first operation request, and the above-mentioned third target metadata server can be a server in a distributed file system that is further determined by the second target metadata server to execute the task corresponding to the first operation request after the front-end device determines the second target metadata server.

[0095] The above-mentioned first target metadata server, second target metadata server, and third target can specifically be physical devices corresponding to physical nodes, or virtual servers running on physical devices. One physical device (hardware device) can run one or more virtual servers. The front-end device can be the same device as the metadata server that performs the task corresponding to the operation request sent by the front-end device, or it can be a device in a distributed file system that is only used to receive operation requests sent by user devices; the configuration device can be the same device as the metadata server that performs the task corresponding to the operation request sent by the front-end device, or it can be a device in a distributed file system that is specifically used to obtain configuration instructions from relevant personnel, set the path index table according to the configuration instructions, and send the path index table to the front-end device when the index table acquisition request of the front-end device is obtained; this application does not limit this.

[0096] The execution principles and interaction processes of the various components in the system embodiment, such as the front-end device, the first target metadata server, the second target metadata server, the third target metadata server and the configuration device, can be found in the description of the following method embodiments.

[0097] Figure 2a A flowchart of a data processing method provided in an exemplary embodiment of the present application is provided. The method is applicable to a front-end device in a distributed file system, including:

[0098] 201. Obtain a first operation request for a first target file, where the first operation request includes path information corresponding to the first target file;

[0099] 202. Using the path index table, according to the longest match principle, match the target directory corresponding to the path information;

[0100] 203. In the path index table, query whether the fragmentation status of the target directory is fragmentable. If so, query the fragmentation type corresponding to the target directory.

[0101] 2031. If the sharding type is a first preset type, determine a first target metadata server for executing the task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type.

[0102] Specifically, the first target file can be a file or a directory, and the type of the above-mentioned first operation request may include any of the following: creation request, deletion request, modification request, query request, etc., which comply with the POSIX (Portable Operating System Interface) standard.

[0103] The path information corresponding to the first target file may include at least one of the following: a file name corresponding to the first target file, and parent directory information of the first target file.

[0104] Specifically, when the path information corresponding to the first target file includes the file name corresponding to the first target file and the parent directory information of the first target file, the path information corresponding to the first target file is its own directory; in addition, when the first target file has no parent directory, the path information corresponding to the first target file may only include the file name of the first target file.

[0105] It should be noted that, in the case where the first operation request is a creation request, since the target server for creating the first target file has not yet been determined, the creation of the first target file on the user device has not been completed, and the file name of the first target file is not stored in the path index table. At this time, the file name of the first target file can be generated by the front-end device according to a preset rule, and the front-end device further determines the first operation request based on the file name of the first target file and the third operation request received from the user device, wherein the third operation request can be triggered by the corresponding operation of the user on the user device side. In addition, the first operation request can also be directly obtained by the front-end device from the user device, which is not limited in this application.

[0106] The target directory is a target directory matched according to the longest matching principle for the path information corresponding to the first target file in the path index table.

[0107] The shard type corresponding to the target directory can be obtained based on the path index table. The path index table stores the target directory and the shard type corresponding to the target directory.

[0108] In some optional embodiments of the present application, the first preset type may be a type with a simpler calculation process corresponding to the first sharding algorithm involved, or when the first sharding algorithm is calculated multiple times, the same type of output is obtained when the input is the same; for example, hash sharding or sharding according to time information.

[0109] Furthermore, the above method also includes step 2032: if the shard type is a second preset type, a second target metadata server is determined based on the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine a third target metadata server that executes the task corresponding to the first operation request; wherein, the second preset type may include random sharding.

[0110] Optionally, for the sharding type of the target directory, when the business applicable to the target directory requires a large number of files or subdirectories under the target directory, the sharding type corresponding to the target directory can be set to hash sharding; when the business applicable to the target directory requires a small number of files or subdirectories under the target directory, the sharding type corresponding to the target directory can be set to random sharding; when the files or subdirectories under the target directory need to be sharded according to the creation time, the sharding type corresponding to the target directory can be set to sharding according to time information.

[0111] Correspondingly, when the first preset type is hash sharding, the first sharding algorithm is a hash algorithm; when the first preset type is sharding according to time information, the first sharding algorithm is a preset algorithm related to time information.

[0112] Furthermore, in the aforementioned step 2031, determining the first target metadata server for executing the task corresponding to the first operation request according to the first sharding algorithm corresponding to the first preset type may include the following steps:

[0113] 20311. Determine a subdirectory corresponding to the target directory from the path information;

[0114] 20312. Calculate a first calculation result using the first sharding algorithm using the subordinate directory, and determine an index path corresponding to the first target file based on the first calculation result and the target directory.

[0115] 20313. Search the path index table for the first target metadata server corresponding to the index path.

[0116] Specifically, the subdirectory described in 20311 above may be the highest-level subdirectory under the target directory. In some optional embodiments of the present application, as shown in Table 1, if the path information included in the first operation request is: / home / user0 / work, where "work" is the file name corresponding to the first target file, then, according to the longest match principle, the target directory corresponding to the matched path information in the path index table is " / home". In this case, querying the path index table reveals that the sharding type corresponding to / home is hash sharding. Based on the path information " / home / user0 / work", the subdirectory of / home is determined to be "user0". Using the subdirectory of / home: "user0", a first calculation result "1" is calculated using the hash algorithm. Based on "1" and " / home", the index path corresponding to the first target file is determined to be: / home / 1. Thus, the path index table finds the metadata server corresponding to the index path / home / 1 with the IP address 192.168.1.11, which is the first target metadata server for the task corresponding to the first operation request.

[0117] Table 1 Example of path index table when the shard type corresponding to the target directory is hash type

[0118] Index item number Index Path Shard Status Sharding Type vMDS MDS-IP 1 / N N / A 0 192.168.1.10 2 / home Y HASH 0 192.168.1.10 3 / home / 1 N N / A 1 192.168.1.11 4 / home / 2 N N / A 2 192.168.1.12 5 / home / 3 N N / A 2 192.168.1.12 ... ... ... ... ... ...

[0119] In Table 1, the index path is used for the front-end device to match the target directory corresponding to the path information corresponding to the first target file based on the index path. The sharding status refers to the sharding status corresponding to the corresponding index path. When the sharding status is "N", it indicates that the corresponding directory (that is, the index path) cannot be sharded. When the sharding status is "Y", it indicates that the corresponding directory can be sharded. When the sharding type is "N / A", it indicates that there is no sharding type. When the sharding type is "RANDOM", it indicates that the sharding type is random sharding. In addition, when the sharding type is "HASH", it indicates that the sharding type is hash sharding. When the sharding type is "TIME", it indicates that the sharding type is sharding according to time information. Table 1 is only an example of a path index table. The content included in the path index table may not be limited to Table 1, and this application does not limit this. The vMDS in Table 1 indicates the virtual metadata server identifier corresponding to its corresponding directory, and MDS-IP indicates the address of the hardware device (that is, metadata server) where the virtual metadata server corresponding to its corresponding directory is located.

[0120] Furthermore, in the aforementioned step 20312, determining the index path corresponding to the first target file from the target directory based on the first calculation result may include:

[0121] The first calculation result is combined with the target directory according to the directory format to obtain an index path corresponding to the first target file.

[0122] For example, based on the aforementioned "1" and " / home", the index path corresponding to the first target file is determined to include:

[0123] Combine "1" with " / home" in directory format (that is, in a format including the " / " symbol) to obtain " / home / 1".

[0124] Furthermore, after determining the first target metadata server, the front-end device sends the first operation request to the first target metadata server, so that the first target metadata server performs corresponding processing based on the first operation request.

[0125] Specifically, the corresponding processing performed by the first target metadata server based on the first operation request is related to the type of the first operation request. For example, if the first operation request is a create request indicating the creation of the first target file, the first target metadata server will create the first target file; if the first operation request is a delete request indicating the deletion of the first target file, the first target metadata server will delete the first target file; if the first operation request is a modify request indicating the modification of the first target file, the first target metadata server will modify the first target file according to the user's instructions; if the first operation request is a query request indicating the query of the first target file, the first target metadata server will display the first target file for the user to query.

[0126] After the front-end device sends the first operation request to the second target metadata server, when the type of the first operation request is a create request, the second target metadata server uses the first operation request to determine a third target metadata server to perform the task corresponding to the first operation request, including:

[0127] According to the file name of the first target file included in the path information, a second calculation result is calculated using a second fragmentation algorithm corresponding to the second preset type, and the third target metadata server is determined according to the second calculation result.

[0128] Specifically, the second target metadata server also stores the shard type corresponding to the target directory. The shard type corresponding to the target directory can be obtained from the metadata of the target directory. The above shard type can be a random type. The second sharding algorithm is a random algorithm. The file name of the first target file can be used as the input of the random function corresponding to the random algorithm to obtain the second calculation result; wherein, the correspondence between the second calculation result and the third target metadata server can be pre-stored in the second target metadata server, and the second target metadata server can determine the third target metadata server based on the correspondence.

[0129] In this application, the front-end device in the distributed file system can realize its function of querying the target directory, as well as the shard status and shard type corresponding to the target directory through the stored path index table, and process some simple calculations to determine the first target metadata server; and after the second target metadata server obtains the first operation request, it can verify the shard type corresponding to the target directory based on its own stored metadata, thereby improving the accuracy of the sharding.

[0130] In some optional embodiments of the present application, a path index table may also be stored in the second target metadata server, and the shard type corresponding to the target directory is determined based on the path index table. This application does not impose any restrictions on this.

[0131] Furthermore, the second target metadata server determines the third target metadata server based on the second calculation result, which may specifically include: determining the identifier of the virtual metadata server corresponding to the target directory based on the second calculation result, and then determining the third target metadata server corresponding to the first target file based on the identifier of the virtual metadata server and a preset correspondence stored in the second target metadata server; wherein the correspondence includes the correspondence between the identifier of the virtual metadata server and the third target metadata server.

[0132] In other optional embodiments of the present application, if the first operation request is a creation request, and the file name of the first target file generated by the front-end device according to the preset rule is "user0", the path index table does not store the file name.

[0133] Specifically, as shown in Table 2, if the first operation request is a create request, and the path information contained in the first operation request is: / home / user0 (here, there is no "user0" in the path index table), then the path index table is used, according to the longest match principle, to match the target directory corresponding to the path information as " / home". At this time, according to the path index table, the shard type corresponding to / home is queried as random sharding. At this time, the second target metadata server corresponding to / home is determined according to / home and the path index table, that is, the metadata server with the address 192.168.1.10, and the first operation request is sent to the second target metadata server.

[0134] Table 2 Example of path index table when the shard type corresponding to the target directory is random

[0135] Index item number Index Path Shard Status Sharding Type vMDS MDS-IP 1 / N N / A 0 192.168.1.10 2 / home Y RANDOM 0 192.168.1.10

[0136] Accordingly, see further Figure 2b As shown, the second target metadata server vMDS-0 may store directory information corresponding to multiple files, vMDS-1 may refer to a third target metadata server (virtual metadata server) determined by a random algorithm, and vMDS-2 may refer to another virtual metadata server. Specifically, the second target metadata server may include metadata for the directory " / " and metadata for subdirectories corresponding to " / ", such as metadata corresponding to "etc," "home," and "tmp." "etc," "home," and "tmp" are directory names corresponding to subdirectories of " / ."

[0137] Furthermore, by creating a new file or a new directory according to the solution of this application, the following can be created: Figure 2c The specific creation method of the directory tree can be found in the above content and will not be repeated here.

[0138] In some other optional embodiments of the present application, after the front-end device completes step 202 and determines the target directory, if the fragmentation status corresponding to the target directory is "N" according to the query of the path index table, that is, the fragmentation type of the target directory is "N / A", that is, the target directory is not fragmented, the method further includes:

[0139] Based on the path index table, a fourth target metadata server is determined to execute the task corresponding to the first operation request, and the first operation request is sent to the fourth target metadata server, causing the fourth target metadata server to execute the task corresponding to the first operation request. The fourth target metadata server is the server directly determined based on the path index table to execute the task corresponding to the first operation request when the front-end device determines that the target directory is not fragmented.

[0140] As shown in Table 3, if the path information contained in the first operation request is: / home / user0 / doc, where "doc" is the file name corresponding to the first target file, then, according to the longest match principle, in the path index table, the target directory corresponding to the matched path information is " / home / user0". At this time, by querying the path index table, it can be determined that the fourth target metadata server that executes the first operation request is the server with the address 192.168.1.11.

[0141] Table 3 Example of path index table when the target directory is not fragmented

[0142] Index item number Index Path Shard Status Sharding Type vMDS MDS-IP 1 / N N / A 0 192.168.1.10 2 / home Y RANDOM 0 192.168.1.10 3 / home / user0 N N / A 1 192.168.1.11 4 / home / user1 N N / A 2 192.168.1.12 5 / home / user2 N N / A 2 192.168.1.12 6 / home / user3 N N / A 1 192.168.1.11 7 / home / user4 N N / A 0 192.168.1.10 ... ... ... ... ... ...

[0143] Furthermore, if the shard type is the first preset type, in order for the first target metadata server to determine that its task is to perform the task corresponding to the first operation request after receiving the first operation request, the method further includes the following steps:

[0144] S1. Adjust the path information in the first operation request according to the first calculation result to obtain a second operation request;

[0145] S2. Send the second operation request to the first target metadata server, so that the first target metadata server performs the task corresponding to the first operation request based on the second operation request.

[0146] Specifically, step S1 may include inserting the first calculation result between the target directory and the directory name of the subordinate directory (the highest-level subdirectory) corresponding to the target directory in the path information according to the directory format, obtaining the intermediate virtual directory information corresponding to the first target file, and generating a second operation request based on the intermediate virtual directory information, wherein the second operation request includes the intermediate virtual directory information. Accordingly, the first target metadata server may determine whether to directly execute the task corresponding to the first operation request based on the second operation request by determining whether the second operation request contains a preset virtual identifier, such as the aforementioned "1".

[0147] For example, as shown in Table 1, if the path information included in the first operation request is: / home / user0 / work, where "work" is the file name corresponding to the first target file, then, according to the longest match principle, the target directory corresponding to the matched path information in the path index table is " / home". At this time, querying the path index table shows that the shard type corresponding to / home is hash sharding. Based on the path information " / home / user0 / work", the subdirectory of / home is determined: "user0". Using the subdirectory of / home: "user0", the hash algorithm is used to calculate a first calculation result "1". "1" is inserted between " / home" and "user0" in " / home / user0 / work" according to the directory format (i.e., a format including the " / " identifier), resulting in " / home / 1 / user0 / work". " / home / 1 / user0 / work" is the intermediate virtual directory information corresponding to the first target file named work. A second operation request is then generated based on this intermediate virtual directory information, so that the first target metadata server executes the task corresponding to the first operation request based on the second operation request.

[0148] It should be noted that the first operation request in this application may also include other information, such as: device identification of the user device, user information, attribute information of the first target file, size information, etc., which is not limited in this application.

[0149] In some other optional embodiments of the present application, the first target metadata server may also determine whether it can execute the task corresponding to the first operation request based on the locally stored metadata, and the present application does not impose any limitation on this.

[0150] Furthermore, before obtaining the first operation request for the first target file, the method further includes the following steps:

[0151] S11. Receive management configuration information, where the management configuration information includes the shard type for the target directory;

[0152] S12. Set the fragmentation type in the path index table based on the management configuration information.

[0153] Specifically, the aforementioned management configuration information can be directly determined based on the configuration instructions sent by relevant personnel, and can also be received from a self-configuration device. In some scenarios, when relevant personnel deploy and plan a distributed file system, and deploy the first target file, or the parent directory of the first target file, they have actually clarified the main purpose of the file, the file mode (scale, distribution, characteristics), and can configure different sharding types for the directories corresponding to different files based on the purpose and file mode of different files. Specifically, when creating a file, you can directly configure the sharding status and sharding type of its corresponding directory. For example: when creating a target directory, directly set the sharding status and sharding type of the target directory.

[0154] Furthermore, the method further comprises the following steps:

[0155] S20, receiving a virtual directory setting instruction for the target directory;

[0156] S21, setting multiple virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

[0157] The plurality of virtual directory information includes the index path.

[0158] Specifically, virtual directory configuration involves sharding the target directory to achieve balanced resource storage. The virtual directory configuration instruction can be sent from the terminal device to the front-end device based on user input. The directory identifiers of the virtual directories included in the target directory, as well as the number of virtual directories, can be user-configured. For example, the directory identifiers of the virtual directories can be 1, 2, 3, or 4, etc.

[0159] Furthermore, in some optional embodiments of the present application, when the first preset type is sharding according to time information, the first sharding algorithm is an algorithm related to time information, such as: modulo calculation, or remainder calculation, etc. Through this algorithm, new files created in the same week can be stored in the same virtual directory related to time information, or new files created on the same day can be stored in the same virtual directory related to time information; specifically, the identifier of the virtual directory can be month information, or year information.

[0160] Furthermore, after the third target metadata server has executed the task corresponding to the first operation request, the method further includes the following steps:

[0161] S31. Obtain update information for the path index table sent by a configuration device;

[0162] S32, using the update information to update the path index table;

[0163] The update information is obtained by the configuration device based on feedback information sent by the first target metadata server or the third target metadata server after executing the task corresponding to the first operation request. The third target metadata server can be the second target metadata server itself or another metadata server.

[0164] Specifically, after executing the task corresponding to the first operation request, the third target metadata server may also send feedback information to the configuration device. This feedback information may be used to instruct the configuration device to update the path index table. For example, if the first operation request is a create request, after the third target metadata server creates the first target file, the feedback information may include: an identifier instructing the configuration device to create a new index item to update the path index table, and the index item to be added corresponding to the first target file.

[0165] Furthermore, if the type of the first operation request is a deletion request, after the third target metadata server deletes the first target file, the feedback information may include: a prompt instructing the configuration device to delete the index item to update the path index table, and the index item to be deleted corresponding to the first target file.

[0166] In other optional embodiments of the present application, if the type of the first operation request is a deletion request, after the third target metadata server deletes the first target file, the feedback information may also be: a prompt instructing the configuration device to delete the index item to update the path index table, and the index item serial number corresponding to the first target file.

[0167] Correspondingly, if the server that executes the task corresponding to the first operation request is the fourth target metadata server, it can also send corresponding feedback information to the configuration device so that the configuration device updates the path index table. The specific processing is similar to that of the third target metadata server and will not be repeated here.

[0168] Furthermore, in some optional embodiments of the present application, the above-mentioned update information may be the path index table after the configuration device updates the path index table stored in it based on the feedback information received from the third target metadata server; the update information may also only include index items that need to be updated, for example: index items that need to be added to the path index table, or index items that need to be deleted from the original path index table.

[0169] Specifically, in the path index table, as shown in Table 1, each row may be referred to as an index item. An index item may specifically include: an index item sequence number, an index path, a shard status corresponding to the index path, a shard type corresponding to the index path, a vMDS corresponding to the index path, and an MDS-IP. If the first operation request is a create request, after the third target metadata server successfully creates the first target file, it may generate a corresponding index item according to a preset rule and send feedback information including the index item to the configuration device. The configuration device then adds the index item to the original path index table based on the received feedback information. In the added index item, the index item sequence number is incremented by 1 from the original maximum sequence number, and the path information corresponding to the first target file is added to the index path column of the index item. The shard status corresponding to the path information corresponding to the first target file is set to a default value of "N," and the shard type corresponding to the path information corresponding to the first target file is set to "N / A." Furthermore, the virtual metadata server identifier corresponding to the path information corresponding to the first target file, as determined in accordance with the aforementioned method, and the address of the corresponding metadata server are added to the vMDS column and the MDS-IP column of the path index table, respectively, to form a new path index table.

[0170] Through the above-mentioned solution of the present application, after a new file or directory is created, the sharding status corresponding to the path information corresponding to the new file is set to the default value "N", which not only meets the user's ease of use but also maintains the locality of the directory tree; so that the parent directory and the child directory in the directory tree are as much as possible on the same hardware device or virtual server, so as to further reduce the overhead of the hardware device or virtual server when synchronizing metadata. In addition, the number of subdirectories of the specified directory stored in it can be monitored by the monitoring program in the front-end device, the first target metadata, the second target metadata server, the third target metadata server or the fourth target metadata server. When the number of its subdirectories exceeds the preset value, the sharding status of the specified target is switched to "Y" and the corresponding sharding type is set to achieve load balancing of each metadata server.

[0171] Furthermore, if the first operation request is a deletion request, after the third target metadata server successfully deletes the first target file, the configuration device can directly delete the index item according to the instructions of the configuration device to update the prompt of the path index table, and delete the corresponding index item according to the index item serial number corresponding to the first target file to obtain a new path index table.

[0172] When the type of the first operation request is a query request or a modification request, the path index table may remain unchanged.

[0173] Furthermore, the first target metadata server includes one or more virtual metadata servers;

[0174] Searching the path index table for the first target metadata server corresponding to the index path includes:

[0175] Determining a target virtual metadata server corresponding to the index path according to the path index table;

[0176] The first target metadata server is determined based on the target virtual metadata server, wherein each virtual metadata server is responsible for providing metadata services for a plurality of directory subtrees and collaborating to provide a unified file system namespace externally.

[0177] Furthermore, in the solution of the present application, the metadata stored in the same virtual metadata server A in the distributed file system can be stored repeatedly in multiple hardware devices. That is, the hardware devices where the virtual metadata server A is located can be multiple. For example, the virtual metadata server A can run in both hardware device B and hardware device C. When hardware device B needs to be further expanded, reduced, or abnormally shut down, the distributed file system can update the pre-stored correspondence between the virtual metadata server A and the hardware device B to the correspondence between the virtual metadata server A and the hardware device C.

[0178] Furthermore, when the first target metadata server or the second target metadata server queries local metadata after obtaining the first operation request and fails to query the target directory, the above method further includes:

[0179] Feedback a query failure prompt to the front-end device so that the front-end device can obtain the latest path index table from the configuration device and re-determine the first target metadata server or the second target metadata server based on the latest path index table.

[0180] Furthermore, the aforementioned S31 and S32 may also be executed only when the directory corresponding to the first target file is the highest-level subdirectory of the target directory, and this application does not impose any limitation on this.

[0181] Furthermore, the distributed file system of the present application can also store file system data corresponding to metadata. File system data and metadata can be provided as separate services or on the same physical node (hardware device). Similarly, file system data can also adopt a sharding strategy. The specific sharding method is similar to the metadata sharding method. In addition, the sharding method of file system data can be stored in the metadata of the corresponding directory, which will not be repeated here. In addition, file system data can also be sharded in other ways. Specifically, the address of the server where the file system data is stored can be adjusted to the offset of the address of the server where the parent directory is located to shard the file system data. The specific sharding method is not limited by this application.

[0182] In addition, in the solution of the present application, the metadata in the distributed file system can be evenly distributed in real time. The sharding strategy is pre-configured and gradually grows with the gradual access of users. While ensuring balanced distribution, metadata 0 migration can be achieved, which is applicable to ultra-large-scale distributed file systems; and it avoids the tedious steps of re-migrating metadata due to uneven metadata distribution in the later stage, and further improves the access efficiency of users to metadata.

[0183] Furthermore, in the solution of the present application, the front-end device in the distributed file system can determine the server that actually processes the task corresponding to the received first operation request based on the index table and the path information of the first target file, combined with the sharding type of the target directory corresponding to the determined path information. This improves the efficiency of determining the server that processes the task corresponding to the first operation request and further improves the efficiency of user access to files or directories. In addition, the sharding solutions provided by the present application are diverse, which can not only meet user usability requirements but also maintain the locality of the directory tree, further reducing the overhead of hardware devices or virtual metadata servers when synchronizing metadata.

[0184] Figure 3 A flowchart of another data processing method provided by an exemplary embodiment of the present application is provided. The method is applicable to a first target metadata server in a distributed file system, comprising the following steps:

[0185] 301. Receive a first operation request sent by a front-end device, where the first operation request includes path information corresponding to the first target file;

[0186] 302. Execute the task corresponding to the first operation request;

[0187] Among them, after the first operation request is obtained by the front-end device, the path index table is used to match the target directory corresponding to the path information according to the longest matching principle. When it is queried in the path index table that the sharding status of the target directory is shardable, and the sharding type corresponding to the target directory is queried as the first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined according to the first sharding algorithm corresponding to the first preset type, and then forwarded to the first target metadata server.

[0188] The specific implementation methods related to this embodiment can be found in the above content and will not be repeated here.

[0189] An exemplary embodiment of the present application further provides a data processing method, which is applicable to a second target metadata server in a distributed file system and may include at least the following steps:

[0190] S01. Receive a first operation request sent by a front-end device, where the first operation request includes path information corresponding to the first target file;

[0191] S02: Determine a third target server that executes a task corresponding to the first operation request using the first operation request.

[0192] Specifically, after receiving the first operation request, the second target metadata server is specifically configured to: use local metadata information and the path information to determine a shard type corresponding to the parent directory information corresponding to the first target file; if the shard type is a second preset type, calculate a second calculation result based on the file name corresponding to the first target file using a second sharding algorithm corresponding to the second preset type; and determine the third target metadata server based on the second calculation result;

[0193] Furthermore, the second target metadata server is further configured to send the first operation request to the third target metadata server, so that the third target metadata server performs the task corresponding to the first operation request;

[0194] Among them, after the front-end device obtains the first operation request, it uses the path index table to match the target directory corresponding to the path information according to the longest match principle, and then queries the path index table to see whether the sharding status of the target directory is shardable. If so, when the sharding type corresponding to the target directory is the second preset type, it is forwarded to the second target metadata server.

[0195] The specific implementation methods related to this embodiment can be found in the above content and will not be repeated here.

[0196] Figure 4 A flowchart of another data processing method provided as an exemplary embodiment of the present application may include at least the following steps:

[0197] 401. The client (located on the front-end device) receives a metadata operation request;

[0198] 402. The client performs a longest match search using the locally cached path index table based on the path name in the metadata operation request to determine a metadata server that can process the metadata operation request.

[0199] 403. The client sends the metadata operation request to a metadata server that can process the current metadata operation request.

[0200] 404. The metadata server determines whether it can determine the target server for executing the task corresponding to the metadata operation request based on the received metadata operation request. If so, the following step 405 is executed. If not, the following steps 4041 to 4042 are executed.

[0201] 4041. Send a prompt message to the client to prompt the client to reconfigure the device to obtain the latest path index table;

[0202] 4042. After receiving the prompt information, the client obtains the latest path index table from the configuration device to update the local path index table;

[0203] 405. The metadata server determines whether the target server is itself. If so, it executes the following step 406. If not, it executes the following step 4051.

[0204] 4051. The metadata server sends the metadata operation request to the target server, so that the target server processes the metadata operation request and responds to the client.

[0205] 406. The metadata server directly processes the metadata operation request and responds to the client.

[0206] Specifically, the metadata operation request may be the aforementioned first operation request; the metadata server may be the aforementioned first target metadata server, second target metadata server, third target metadata server, or fourth target metadata server.

[0207] Furthermore, when the metadata operation request is a request to create or delete a file, in this method, after the target server creates or deletes the file, it also includes updating its locally stored path index table and sending the updated path index table to the configuration device.

[0208] The specific implementation of this embodiment can be found in the above content and will not be repeated here.

[0209] Figure 5 This is a schematic diagram of the structure of another data processing device provided in an exemplary embodiment of the present application. The device is specifically applicable to a front-end device in a distributed file system and may include: an acquisition module 51, a matching module 52, a query module 53, and a determination module 54; wherein:

[0210] An acquisition module 51 is configured to acquire a first operation request for a first target file, wherein the first operation request includes path information corresponding to the first target file;

[0211] A matching module 52 is configured to use the path index table to match the target directory corresponding to the path information according to the longest match principle;

[0212] A query module 53 is configured to query the path index table to determine whether the target directory is fragmentable, and if so, query the fragment type corresponding to the target directory;

[0213] The determination module 54 is configured to determine, if the sharding type is a first preset type, according to a first sharding algorithm corresponding to the first preset type, a first target metadata server for executing the task corresponding to the first operation request.

[0214] Optionally, the determination module 54 is also used to: if the shard type is a second preset type, determine the second target metadata server based on the target directory and the path index table, and send the first operation request to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that performs the task corresponding to the first operation request.

[0215] Optionally, when the determining module 54 is used to determine the first target metadata server for executing the task corresponding to the first operation request according to the first sharding algorithm corresponding to the first preset type, it is specifically used to:

[0216] Determine the subdirectory corresponding to the target directory from the path information;

[0217] Using the lower-level directory, calculate a first calculation result using the first sharding algorithm;

[0218] Determining an index path corresponding to the first target file according to the first calculation result and the target directory;

[0219] The first target metadata server corresponding to the index path is searched from the path index table.

[0220] Optionally, the path information includes a file name of the first target file. When the type of the first operation request is a create request, the second target metadata server, when using the first operation request to determine a third target metadata server that executes a task corresponding to the first operation request, is specifically configured to:

[0221] According to the file name, a second calculation result is calculated using a second sharding algorithm corresponding to the second preset type, and the third target metadata server is determined according to the second calculation result.

[0222] Optionally, the device is further used for:

[0223] Adjust the path information in the first operation request according to the first calculation result to obtain a second operation request;

[0224] The second operation request is sent to the first target metadata server, so that the first target metadata server performs the task corresponding to the first operation request based on the second operation request.

[0225] Optionally, before the acquisition module 51 acquires the first operation request for the first target file, the apparatus is further configured to:

[0226] receiving management configuration information, wherein the management configuration information includes the shard type for the target directory;

[0227] The fragmentation type is set in the path index table based on the management configuration information.

[0228] Optionally, the device is further used for:

[0229] receiving a virtual directory setting instruction for the target directory;

[0230] Setting multiple virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

[0231] The plurality of virtual directory information includes the index path.

[0232] Optionally, when the directory corresponding to the first target file is the highest-level subdirectory of the target directory, and the third target metadata server has executed the task corresponding to the first operation request, the apparatus is further configured to:

[0233] Obtaining update information for the path index table sent by the configuration device;

[0234] Updating the path index table using the update information;

[0235] The update information is obtained by the configuration device according to feedback information sent by the third target metadata server after executing the task corresponding to the first operation request.

[0236] Optionally, the first target metadata server includes multiple virtual metadata servers;

[0237] When the determining module 54 is configured to search the path index table for the first target metadata server corresponding to the index path, it is specifically configured to:

[0238] Determining a target virtual metadata server corresponding to the index path according to the path index table;

[0239] The first target metadata server is determined based on the target virtual metadata server.

[0240] Optionally, the first preset type includes hash sharding or sharding according to time information; the second preset type includes random sharding.

[0241] Optionally, the type of the first operation request includes any one of the following: a create request, a delete request, a modify request, and a query request.

[0242] The specific implementation of this embodiment can be found in the above content and will not be repeated here.

[0243] Figure 6 This is a schematic diagram of the structure of another data processing device provided by an exemplary embodiment of the present application. The device is specifically applicable to a first target metadata server in a distributed file system, including: a receiving module 61 and an execution module 62; wherein:

[0244] The receiving module 61 is configured to receive a first operation request sent by a front-end device, wherein the first operation request includes path information corresponding to the first target file;

[0245] An execution module 62, configured to execute the task corresponding to the first operation request;

[0246] Among them, after the first operation request is obtained by the front-end device, the path index table is used to match the target directory corresponding to the path information according to the longest matching principle. When it is queried in the path index table that the sharding status of the target directory is shardable, and the sharding type corresponding to the target directory is queried as the first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined according to the first sharding algorithm corresponding to the first preset type, and then forwarded to the first target metadata server.

[0247] The specific implementation of this embodiment can be found in the above content and will not be repeated here.

[0248] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the data processing methods provided in the above embodiments.

[0249] The following describes the specific implementation process of the solution in the embodiment of the present application when the types of the first operation request are creation request, deletion request, modification request and query request, respectively, in combination with specific scenarios:

[0250] If the first target file is named "work", its parent directory is " / home / user0";

[0251] For the first operation request which is a request to create the first target file, the first operation request may include the parent directory information of the first target file (for example, / home / user0), and the file name corresponding to the first target file (for example, work) generated by the front-end device / or user device based on the creation request according to preset rules. The path information corresponding to the first target file is: " / home / user0 / work".

[0252] After the front-end device receives the first operation request, if it uses the path index table stored in its own storage and matches the target directory " / home" according to the longest match principle, and in the path index table, it is found that the sharding status of the target directory " / home" is shardable, and the sharding type corresponding to " / home" is hash sharding, then it uses the subordinate directory of / home: "user0" to calculate the first calculation result "1" through the hash algorithm; then, based on "1" and " / home", it determines the index path corresponding to the first target file: / home / 1, and thus finds the first target metadata server corresponding to the index path ( / home / 1) through the path index table; and sends the first operation request to the first target metadata server, so that the first target metadata server executes the operation of creating the first target file named "work";

[0253] If the front-end device uses the path index table stored in its own storage and matches the target directory " / home / user0" according to the longest match principle, and finds that the shard type corresponding to " / home / user0" is random sharding, then the second target metadata server is determined based on the target directory " / home / user0" and the path index table, and the first operation request is sent to the second target metadata server. The second target metadata is used to calculate work through a random algorithm to obtain a corresponding server identifier. The third target metadata server is determined based on the server identifier, and the first operation request is sent to the third target metadata server, so that the third target metadata server executes the operation of creating the first target file named "work";

[0254] If the front-end device uses the path index table stored in its own storage, according to the longest match principle, the target directory matched is " / home / user0", and the shard type corresponding to " / home / user0" is found to be non-shardable, then the fourth target metadata server is determined based on the target directory " / home / user0" and the path index table, and the first operation request is sent to the fourth target metadata server, and the fourth target metadata server executes the operation of creating the first target file named "work".

[0255] Application scenario 2: When the first operation request is to delete the first target file named "work" whose parent directory is " / home / user0", the path information corresponding to the first target file is: " / home / user0 / work";

[0256] If the front-end device uses the path index table stored in its own storage and matches the target directory " / home" according to the longest match principle, and in the path index table, it is found that the sharding status of the target directory " / home" is shardable, and the sharding type corresponding to " / home" is hash sharding, then the lower-level directory of / home: "user0" is used to calculate the first calculation result "1" through the hash algorithm; then, based on "1" and " / home", the index path corresponding to the first target file is determined: / home / 1, so as to find the first target metadata server corresponding to the index path ( / home / 1) through the path index table; and send the first operation request to the first target metadata server (that is, the first target file named work is stored in the first target metadata server), so that the first target metadata server executes the operation of deleting the first target file named "work";

[0257] If the front-end device uses the path index table stored in itself, according to the longest match principle, the target directory matched is " / home / user0", and in the path index table, it is found that the fragmentation status of the target directory " / home / user0" is not fragmentable, then according to the path index table, the fourth target metadata server is determined (that is, the first target file named work is stored in the fourth target metadata server), so that the fourth target metadata server deletes the first target file.

[0258] Application scenario three: When the first operation request is to modify the first target file named "work" in the parent directory " / home / user0", the path information corresponding to the first target file is: " / home / user0 / work";

[0259] If the front-end device uses the path index table stored in its own storage and matches the target directory " / home" according to the longest match principle, and in the path index table, it is found that the sharding status of the target directory " / home" is shardable, and the sharding type corresponding to " / home" is hash sharding, then the lower-level directory of / home: "user0" is used to calculate the first calculation result "1" through the hash algorithm; then, based on "1" and " / home", the index path corresponding to the first target file is determined: / home / 1, so as to find the first target metadata server corresponding to the index path ( / home / 1) through the path index table; and send the first operation request to the first target metadata server (that is, the first target file named work is stored in the first target metadata server), so that the first target metadata server executes the operation of modifying the first target file named "work";

[0260] If the front-end device uses the path index table stored in itself, according to the longest match principle, the target directory matched is " / home / user0", and in the path index table, it is found that the fragmentation status of the target directory " / home / user0" is not fragmentable, then according to the path index table, the fourth target metadata server is determined (that is, the first target file named work is stored in the fourth target metadata server), so that the fourth target metadata server executes the operation of modifying the first target file.

[0261] Application scenario three: When the first operation request is to query the first target file named "work" in the parent directory " / home / user0 / ", the path information corresponding to the first target file is: " / home / user0 / work";

[0262] If the front-end device uses the path index table stored in its own storage and matches the target directory " / home" according to the longest match principle, and in the path index table, it is found that the sharding status of the target directory " / home" is shardable, and the sharding type corresponding to " / home" is hash sharding, then the lower-level directory of / home: "user0" is used to calculate the first calculation result "1" through the hash algorithm; then, based on "1" and " / home", the index path corresponding to the first target file is determined: / home / 1, so as to find the first target metadata server corresponding to the index path ( / home / 1) through the path index table; and send the first operation request to the first target metadata server (that is, the first target file named work is stored in the first target metadata server), so that the first target metadata server executes the operation of modifying the first target file named "work";

[0263] If the front-end device uses the path index table stored in its own storage, according to the longest match principle, the target directory matched is " / home / user0", and in the path index table, it is found that the fragmentation status of the target directory " / home / user0" is not fragmentable, then according to the path index table, the fourth target metadata server is determined (that is, the first target file named work is stored in the fourth target metadata server), so that the fourth target metadata server displays the first target file according to the user instruction.

[0264] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0265] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0266] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: Applicable to front-end devices in distributed file systems, including: Obtaining a first operation request for a first target file, where the first operation request includes: path information corresponding to the first target file; Using the path index table, according to the longest match principle, the target directory corresponding to the path information is matched; In the path index table, query whether the fragmentation status of the target directory is fragmentable, and if so, query the fragmentation type corresponding to the target directory; If the sharding type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type; the first preset type includes hash sharding or sharding according to time information; If the sharding type is the second preset type, the second target metadata server is determined based on the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that executes the task corresponding to the first operation request; the second preset type includes random sharding.

2. The method according to claim 1, characterized in that Determining, according to a first sharding algorithm corresponding to the first preset type, a first target metadata server for executing the task corresponding to the first operation request includes: Determine the subdirectory corresponding to the target directory from the path information; Using the lower-level directory, calculate a first calculation result using the first sharding algorithm; Determining an index path corresponding to the first target file according to the first calculation result and the target directory; The first target metadata server corresponding to the index path is searched from the path index table.

3. The method according to claim 1, characterized in that The path information includes the file name of the first target file. When the type of the first operation request is a create request, the second target metadata server uses the first operation request to determine a third target metadata server that executes a task corresponding to the first operation request, including: According to the file name, a second calculation result is calculated using a second sharding algorithm corresponding to the second preset type, and the third target metadata server is determined according to the second calculation result.

4. The method according to claim 2, characterized in that The method further comprises: Adjust the path information in the first operation request according to the first calculation result to obtain a second operation request; The second operation request is sent to the first target metadata server, so that the first target metadata server performs the task corresponding to the first operation request based on the second operation request.

5. The method according to claim 1, wherein Before obtaining the first operation request for the first target file, the method further includes: receiving management configuration information, wherein the management configuration information includes the shard type for the target directory; The fragmentation type is set in the path index table based on the management configuration information.

6. The method according to claim 2, characterized in that The method further comprises: receiving a virtual directory setting instruction for the target directory; Setting multiple virtual directory information corresponding to the target directory according to the virtual directory setting instruction; The plurality of virtual directory information includes the index path.

7. The method according to claim 1, characterized in that When the directory corresponding to the first target file is the highest-level subdirectory of the target directory, and the third target metadata server has executed the task corresponding to the first operation request, the method further includes: Obtaining update information for the path index table sent by the configuration device; Updating the path index table using the update information; The update information is obtained by the configuration device according to feedback information sent by the third target metadata server after executing the task corresponding to the first operation request.

8. The method according to claim 2, characterized in that The first target metadata server includes a plurality of virtual metadata servers; Searching the path index table for the first target metadata server corresponding to the index path includes: Determining a target virtual metadata server corresponding to the index path according to the path index table; The first target metadata server is determined based on the target virtual metadata server.

9. The method according to claim 1, characterized in that The type of the first operation request includes any one of the following: a create request, a delete request, a modify request, and a query request.

10. A data processing method, characterized in that: A first target metadata server applicable to a distributed file system, comprising: Receive a first operation request sent by a front-end device, where the first operation request includes: path information corresponding to a first target file; Execute the task corresponding to the first operation request; Among them, after the front-end device obtains the first operation request, it uses the path index table to match the target directory corresponding to the path information according to the longest match principle. When it is queried in the path index table that the sharding status of the target directory is shardable and the sharding type corresponding to the target directory is a first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined according to the first sharding algorithm corresponding to the first preset type, and then forwarded to the first target metadata server; the first preset type includes hash sharding or sharding according to time information; if the sharding status of the target directory is shardable and the sharding type is the second preset type, the second target metadata server is determined according to the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that executes the task corresponding to the first operation request; the second preset type includes random sharding.

11. A data processing device, characterized in that: Applicable to front-end devices in distributed file systems, including: An acquisition module, configured to acquire a first operation request for a first target file, wherein the first operation request includes: path information corresponding to the first target file; A matching module is used to match the target directory corresponding to the path information using the path index table according to the longest match principle; A query module, configured to query the path index table to determine whether the target directory is fragmentable, and if so, to determine the fragment type corresponding to the target directory; A determination module is used to determine, if the sharding type is a first preset type, a first target metadata server that executes the task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type; the first preset type includes hash sharding or sharding according to time information; if the sharding type is a second preset type, a second target metadata server is determined based on the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine a third target metadata server that executes the task corresponding to the first operation request; the second preset type includes random sharding.

12. A data processing device, characterized in that: A first target metadata server applicable to a distributed file system, comprising: A receiving module, configured to receive a first operation request sent by a front-end device, wherein the first operation request includes: path information corresponding to a first target file; an execution module, configured to execute the task corresponding to the first operation request; Among them, after the front-end device obtains the first operation request, it uses the path index table to match the target directory corresponding to the path information according to the longest match principle. When it is queried in the path index table that the sharding status of the target directory is shardable and the sharding type corresponding to the target directory is a first preset type, the first target metadata server that executes the task corresponding to the first operation request is determined according to the first sharding algorithm corresponding to the first preset type, and then forwarded to the first target metadata server; the first preset type includes hash sharding or sharding according to time information; if the sharding status of the target directory is shardable and the sharding type is the second preset type, the second target metadata server is determined according to the target directory and the path index table, and the first operation request is sent to the second target metadata server, so that the second target metadata server uses the first operation request to determine the third target metadata server that executes the task corresponding to the first operation request; the second preset type includes random sharding.

13. A distributed file system, characterized in that: include: Front-end equipment, configuration equipment and at least one server; wherein: A front-end device, configured to obtain a first operation request for a first target file, wherein the first operation request includes: path information corresponding to the first target file; using a path index table, according to a longest match principle, matching a target directory corresponding to the path information; inquiring in the path index table whether the sharding status of the target directory is shardable, and if so, inquiring about the sharding type corresponding to the target directory; if the sharding type is a first preset type, determining a first target metadata server that executes a task corresponding to the first operation request according to a first sharding algorithm corresponding to the first preset type; the first preset type includes hash sharding or sharding according to time information; if the sharding type is a second preset type, determining a second target metadata server according to the target directory and the path index table, and sending the first operation request to the second target metadata server, so that the second target metadata server determines a third target metadata server that executes the task corresponding to the first operation request by using the first operation request; the second preset type includes random sharding; A first target metadata server among the at least one server is configured to execute a task corresponding to the first operation request; The configuration device is used to obtain configuration instructions from relevant personnel, set the path index table according to the configuration instructions, and send the path index table to the front-end device when obtaining an index table acquisition request from the front-end device.

Images