File processing method, file system and readable storage medium

Abstract

The invention discloses a file processing method, a file system and a readable storage medium. The method is applied to a system comprising a private client, a master file server and fragmented file servers, the master file server segments a target file and distributes obtained N fragmented files to M file servers for management, and the method comprises the following steps: the private client performs data distribution on the fragmented files according to a first data layout table, when it is determined that a first fragment identifier of to-be-written first data is larger than a first maximum fragment identifier, first fragment creation information is sent to the master file server, so that the master file server creates a first fragment, the maximum fragment identifier included in the second data layout table is updated to be the first fragment identifier, and the first fragment is written into the second data layout table; and sending the updated second data layout table to the first private client. According to the fragment creation process, the accuracy of the data layout table can be ensured by updating the data layout table, so that the accuracy of file processing operations such as attribute acquisition executed by a subsequent private client is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, in particular to a file processing method, a file system and a readable storage medium. BACKGROUND

[0002] In a distributed file system, in the case that multiple user devices access the same file, the access and update of data by the multiple user devices need to be processed by a home metadata server (MDS), wherein the home MDS is a specific MDS that stores file metadata and the like, and is used to manage the metadata (for example, creation time, modification time, file data volume, etc.) and data of the file in the storage device, so as to ensure the metadata consistency and correctness of the file system. However, the processing pressure of a single home MDS is large, and the performance of concurrent read-write operations of multiple user devices is poor.

[0003] At present, by means of data sharding, large file data segments can be divided into multiple MDSs according to certain rules for management, so as to reduce the processing pressure of a single home MDS, and thus the performance of concurrent read-write operations of multiple user devices can be improved. However, with the improvement of the performance of concurrent read-write operations, the number of MDSs that need to be coordinated for file semantic operations (for example, data read-write, file truncation or file attribute acquisition, etc.) also increases, which is not conducive to the efficient and accurate implementation of file semantic operations, and the simultaneous execution of data read-write operations and file semantic operations will cause data inconsistency. SUMMARY

[0004] The present application discloses a file processing method, a file system and a readable storage medium. A distributed private client (DPC), a master file server (MDS) and a sharded file server (MDS) cooperatively process different file semantic operations through a data layout table (maptable) including private data such as file data volume owner identification (sizeowner), version number (maptable version) and bitmap (maptable bitmap), implement the management of files, can reduce the limitation on the number of sharded file servers, improve the performance of file semantic operations such as truncate operation or getattr operation in the case of improving the performance of data concurrent read-write, and ensure the consistency of file data in the process of file semantic operations.

[0005] In a first aspect, the present application provides a file semantic processing method, which is applied to a distributed file system (DFS) including at least one private client and a plurality of file servers, the plurality of file servers including a master file server and a plurality of shard file servers, wherein the at least one private client includes a first private client, the master file server splits a target file of the first private client into N shard files, and the N shard files are managed by M file servers, wherein N is a positive integer greater than 1, and M is greater than or equal to N. The method includes: determining, by the first private client according to a data offset address of first data to be written and a length of the first data, a first shard identifier to which the first data is to be written, and sending, by the first private client, first shard creation information to the master file server in a case where a first maximum shard identifier included in a cached first data layout table is less than the first shard identifier. Creating, by the master file server according to the first shard creation information and a stored second data layout table, the first shard, and updating a second maximum shard identifier in the second data layout table to the first shard identifier, wherein the first shard identifier is greater than the second maximum shard identifier. Sending, by the master file server, the updated second data layout table to the first private client, so that the first private client sends the first data to a first shard file server to which the first shard belongs according to the updated second data layout table.

[0006] In summary, the private client determines whether to send shard creation information according to the maximum shard identifier of the cached data layout table, and the master file server updates the target file modification time, the data amount of the target file and other data in the process of creating the shard, which can ensure the accuracy of the data layout table, and in the case where the private client performs an attribute acquisition operation subsequently, the file attribute information is determined according to the data layout table updated by the master file server, so as to determine the accurate file size after the first shard is created.

[0007] Exemplarily, before the first private client creates the first shard to the master file server according to the cached first data layout table and the first shard identifier, the method further comprises: the first private client determines that the sum of the data amount of the first data to be written and the second data already written is greater than a split threshold, and sends file split information to the master file server, wherein the split threshold is the maximum data amount managed by the master file server; the master file server creates a first data layout table according to the file split information, and sends shard file creation information to one or more shard file servers, wherein the first data layout table comprises a first maximum shard identifier and a first file data amount owner identifier, and the first file data amount owner identifier is used to indicate the second shard file server to which the shard corresponding to the first maximum shard identifier belongs; and the master file server sends the first data layout table to the first private client after receiving the shard file creation response sent by each of the plurality of shard file servers.

[0008] In the above process, the master file server splits the file upon receiving the file split information, and sends shard file creation information to one or more shard file servers, so that a larger file can be managed by multiple shard file servers, and the performance of concurrent read-write operations of multiple private clients can be improved. In the case where multiple shard file servers are needed to manage a file, the master file server creates a data layout table, and the file data amount owner identifier and other information included in the data layout table can be used to determine file size and other attribute information.

[0009] Exemplarily, the master file server updates the first file data amount owner identifier included in the second data layout table to a second file data amount owner identifier, wherein the second file data amount owner identifier is used to indicate the first shard file server to which the first shard belongs.

[0010] The file data amount owner identifier can be used to indicate the shard file server to which the maximum shard identifier belongs, and in the execution process of the subsequent attribute acquisition operation, the accurate file size can be determined according to the file data amount owner identifier.

[0011] Exemplarily, in the process of creating the first shard, the method further comprises: the master file server sends identifier clearing information to the second shard file server; the second shard file server clears the file data amount owner identifier according to the identifier clearing information, and sends first cache invalidation information to the plurality of private clients through the distributed lock server, so that the shard file metadata included in the second shard file server cached on the plurality of private clients is invalidated, and the shard file metadata comprises the data amount of the plurality of shards included in the second shard file server.

[0012] In the above process, the private client clears the shard file metadata included in the second shard file server, which can avoid the private client performing an attribute acquisition operation to calculate the file size before the first shard is created according to the data amount of the plurality of shards included in the second shard file server during the first shard creation process, thereby avoiding inaccurate file size calculation.

[0013] Exemplarily, the method further includes: the master file server sending identification setting information to the first shard file server; the first shard file server setting the second file data amount owner identification according to the identification setting information, and updating the first shard file data amount included in the first shard file server to the second shard file data amount, wherein the second shard file data amount is determined according to the first shard identification and the data amount of each of the plurality of shards included in the file; and the first shard file server sending the second cache invalidation information to the plurality of private clients through the distributed lock server, so as to invalidate the second shard file metadata corresponding to the first shard file server cached on the plurality of private clients.

[0014] In the above process, the first shard file server updates the data amount of the plurality of shards included in the first shard after the first shard is created, and invalidates the data amount of the plurality of shards before the first shard is created cached in the private client, so as to avoid the problem that the private client cannot accurately calculate the file size after the first shard is created, by determining the file size according to the cached data amount of the plurality of shards included in the first shard file server before the first shard is created after determining the second file data amount owner identification after the first shard is created. Exemplarily, the at least one private client further includes a second private client, and the method further includes: the second private client determining a second shard identification of third data to be written, wherein the second shard identification is obtained based on a data offset address of the third data and a length of the third data; the second private client sending a second shard creation message to the master file server according to the cached third data layout table and the second shard identification, the third data layout table including a third maximum shard identification and a first bit, the third maximum shard identification being greater than or equal to the second shard identification, wherein a bit value of the first bit indicates that the data included in the shard corresponding to the second shard identification is invalid; the master file server sending data clearing information to a third shard file server to which the second shard belongs according to the second shard creation information and a stored fourth data layout table, wherein the fourth data layout table includes a fourth maximum shard identification and a second bit, the fourth maximum shard identification being greater than or equal to the second shard identification, and a bit value of the second bit indicating that the data included in the shard corresponding to the second shard identification is invalid; and the third shard file server clearing the data included in the second shard in response to the data clearing information, and sending a data clearing response to the master file server.

[0015] In the above process, the master file server does not need to create a new shard according to the second shard creation information in a case where it is determined that the second shard exists but the data included in the second shard is invalid, but only needs to clear the dirty data included in the second shard to make the space for writing the third data exist in the shard, simplifying the operation process, and by indicating whether the data is valid through the bit, it can be avoided that the private client writes data into the shard with dirty data, ensuring the performance of data write operation and avoiding the case of failed data write. Illustratively, the at least one private client further includes a third private client, and the method further includes: the third private client determines a third shard identifier of fourth data to be written, the third shard identifier being obtained based on a data offset address of the fourth data and a length of the fourth data; the third private client sends a version number of a fifth data layout table and the third shard identifier to a fourth shard file server, the fifth data layout table including a fifth maximum shard identifier and a second bit, the fifth maximum shard identifier being greater than or equal to the third shard identifier, the bit value of the second bit indicating that the data included in the shard corresponding to the third shard identifier is valid, and the version number of the fifth data layout table being the version number of the fifth data layout table before the file truncation operation is performed on the file; the master file server sends a version number of a sixth data layout table to the fourth shard file server, wherein the version number of the sixth data layout table is the version number after the file truncation operation is performed on the file, and the version number of the sixth data layout table is greater than the version number of the fifth data layout table; the fourth shard file server determines that the version number of the fifth data layout table and the version number of the sixth data layout table are not the same, and returns a write data error to the third private client.

[0016] In the above process, the private client sends the version number of the cached data layout table to the shard file server including the third shard in a case where it is determined that the third shard exists and the data included in the third shard is valid, so that the shard file server obtains the latest version number from the master file server in a case where the version number does not exist, and determines the occurrence of the file truncation operation in a case where the version numbers are not the same. At this time, the data included in the third shard is not necessarily valid, and in order to ensure the consistency of the data, the data write operation is not performed.

[0017] Illustratively, after the private client determines the first shard identifier of the first data to be read, the private client determines that the first shard exists and the data included in the first shard is valid according to the cached fifth data layout table, and sends the version number of the fifth data layout table and the first shard identifier to the first shard file server to which the first shard belongs. The first shard file server receives the latest version number of the sixth data layout table sent by the master file server, and returns a read data error in a case where it is determined that the version number of the sixth data layout table and the version number of the fifth data layout table are not the same.

[0018] In the above process, the slice file server can determine, through the version number, that the read operation and the file truncation operation are concurrently executed, and the data included in the first slice may not be valid. In the case where the data included in the first slice is affected by the file truncation operation and becomes invalid, the above method can avoid that the private client reads dirty data in the first slice.

[0019] Exemplarily, the at least one private client further includes a fourth private client, and the method further includes: the fourth private client sending a file truncation operation to the master file server; the master file server sending version number clearing information to one or more slice file servers corresponding to the file truncation operation according to the file truncation operation; and the master file server updating a version number of a seventh data layout table in the cache to a version number of a sixth data layout table according to the file truncation operation, the version number of the seventh data layout table being a version number before the file truncation operation is performed on the file.

[0020] The version number is used to indicate the number of file truncation operations, and is used to determine whether there is a parallel file truncation operation in the process in which the private client performs a data read / write operation, so as to avoid problems such as reading dirty data or writing failure caused by the slice file server directly performing a data read / write operation.

[0021] Exemplarily, the master file server updates the seventh data layout table according to the file truncation operation, and the updating specifically includes: updating a third file data amount owner identifier included in the seventh data layout table to a fourth file data amount owner identifier included in the sixth data layout table, wherein the third file data amount owner identifier is used to indicate a fifth slice file server to which a slice corresponding to a seventh maximum slice identifier belongs, and the fourth file data amount owner identifier is used to indicate a sixth slice file server to which a slice corresponding to a sixth maximum slice identifier belongs; and modifying a first bit value of one or more fourth bits included in the seventh data layout table to a second bit value, wherein the fourth bit is used to indicate a slice corresponding to the first file truncation operation, and the first bit value indicates that data included in the slice is valid, and the second bit value indicates that the data included in the slice is invalid.

[0022] Updating the data layout table can more accurately manage the file, determine the file data amount owner identifier after the file truncation operation, and be used to determine an accurate file size after the file truncation operation in a subsequent attribute acquisition operation, and modifying the bit values of the plurality of bits can determine whether the data included in the slice is valid in the case where slice creation information is subsequently received, so as to ensure the performance of the data read / write operation.

[0023] Exemplarily, the first private client determines the first file attribute information according to the first attribute obtaining operation, the cached eighth data layout table, and the cached file server included shard file data amount, wherein the first private client cached file server included shard file data amount is valid, and the first file attribute information includes at least one of the following: target file data amount, target file creation time, and target file modification time.

[0024] Since the data layout table and the file server included shard file data amount are cached in the private client, the file size can be directly calculated in the case of valid data, the private client does not need to send shard file data amount obtaining information to each file server, the performance of the attribute obtaining operation is improved, and the number of file servers does not need to be limited in order to improve the performance of the attribute obtaining operation.

[0025] Exemplarily, the first private client determines the second file attribute information according to the second attribute obtaining operation, the cached ninth data layout table, and the cached file server included shard file data amount, sends data layout table obtaining information to the master file server, and sends data amount obtaining information to the first shard file server, wherein the first private client cached file server included shard file data amount is invalid; the first private client determines the second file attribute information according to the second data layout table sent by the master file server and the target shard file data amount sent by the first shard file server, wherein the second file attribute information includes at least one of the following: target file data amount, target file creation time, and target file modification time.

[0026] The above process is the case that the first shard file server sends the included multiple shard data amounts to the first private client in the case of determining that the self-included file data amount owner identifier, the private client only needs to send shard file data amount obtaining information to the first shard file server, and does not need to send shard file data amount obtaining information to all file servers, which can save the calculation overhead.

[0027] Exemplarily, the first private client sends data layout table obtaining information to the master file server and sends data amount obtaining information to the first file server according to the third attribute obtaining operation, the cached tenth data layout table, and the cached data amount of the plurality of file servers each including a shard file, wherein the first private client caches the data amount of the first shard file server including a shard file is invalid; the first private client obtains the eleventh data layout table from the master file server in the case that the first file server does not include the file data amount owner identifier, determines a new file data amount owner identifier according to the eleventh data layout table, sends the shard file data amount to the file server indicated by the new file data amount owner identifier, and determines the accurate file size according to the shard file data amount.

[0028] Compared with the current file attribute information obtaining operation, the private client needs to send data obtaining information to the master file server and each shard file server including a shard, and determines the file metadata and the accurate file data amount by adding the shard data amount sent by each of the plurality of shard file servers. In the case that the plurality of shards in the file are managed by the plurality of shard file servers, the performance of the private client performing concurrent data read-write operation is improved, but the file attribute information obtaining operation needs to send file data amount obtaining information to more shard file servers, and the computing resource consumption is large and the performance is poor. In the specific process of the above file attribute obtaining operation, the private client can determine the file data amount according to the cached file metadata, shard size, split threshold, number of shard file servers, and shard data amount corresponding to the shard file server sent by the master file server and the shard file server having the file data amount owner identifier, without sending file data amount obtaining information to the plurality of shard file servers, so as to determine the accurate file data amount, reduce the consumption of computing resources, and ensure the performance of the file attribute obtaining operation in the case that the number of shard file servers increases.

[0029] Exemplarily, the method further includes: the first private client determines a fifth shard identifier of the fifth data to be read, the fifth shard identifier is obtained based on a data offset address of the fifth data and a length of the fifth data, and in the case that the first private client determines that the fifth shard identifier is greater than the first maximum shard identifier in the cached first data layout table according to the cached first data layout table and the fifth shard identifier, it is determined that the fifth shard does not exist, and a read data error is returned. In the case that the first private client determines that the data included in the fifth shard is invalid according to the bit value of the bit indicating the fifth shard in the cached first data layout table, a read data error is still returned.

[0030] Exemplarily, the method further comprises: the first private client determining a sixth shard identifier of the sixth data to be read, the sixth shard identifier being obtained based on a data offset address of the sixth data and a length of the sixth data, in a case where, according to the cached first data layout table and the sixth shard identifier, it is determined that the sixth shard identifier is greater than a first maximum shard identifier in the first data layout table, it is determined that the sixth shard exists, and in a case where, according to a bit value of a bit indicating the sixth shard, it is determined that the data included in the sixth shard is valid, the first private client sends the sixth shard identifier and a version number of the first data layout table to a shard file server to which the sixth shard belongs, so as to ensure normal reading of the sixth data.

[0031] In a second aspect, the present application provides a file system, which comprises at least one private client and a plurality of file servers, the plurality of file servers comprising a master file server and a plurality of shard file servers, the at least one private client comprising a first private client, the master file server splitting a target file of the first private client into N shard files, the N shard files being managed by M file servers, wherein N is a positive integer greater than 1, and M is greater than or equal to N; the first private client is configured to determine a first shard identifier of first data to be written, the first shard identifier being obtained based on a data offset address of the first data and a length of the first data; the first private client is further configured to send first shard creation information to the master file server according to a cached first data layout table and the first shard identifier; the master file server is configured to create the first shard according to the first shard creation information and a stored second data layout table, wherein the first shard identifier is greater than a second maximum shard identifier in the second data layout table, and the master file server is further configured to update the second maximum shard identifier included in the second data layout table to the first shard identifier, and to send the updated second data layout table to the first private client, so that the first private client sends the first data to a first shard file server to which the first shard belongs according to the updated second data layout table.

[0032] Exemplarily, the first private client is further configured to determine that a sum of a data amount of the first data to be written and a data amount of the second data that has been written is greater than a split threshold, and to send file split information to the master file server, wherein the split threshold is a maximum data amount managed by the master file server; the master file server is further configured to create a first data layout table according to the file split information, and to send shard file creation information to one or more shard file servers, wherein the first data layout table comprises a first maximum shard identifier and a first file data amount owner identifier, the first file data amount owner identifier being used to indicate a second shard file server to which a shard corresponding to the first maximum shard identifier belongs; and the master file server is further configured to send the first data layout table to the first private client after receiving shard file creation responses sent by the plurality of shard file servers respectively.

[0033] Exemplarily, the master file server is further configured to update the first file data amount owner identifier included in the second data layout table to a second file data amount owner identifier, where the second file data amount owner identifier is used to indicate the first shard file server to which the first shard belongs.

[0034] Exemplarily, the master file server is further configured to send identifier clearing information to the second shard file server; the second shard file server is configured to clear the file data amount owner identifier according to the identifier clearing information, and send first cache invalidation information to the plurality of private clients through the distributed lock server, so that the second shard file metadata included in the first shard file server and cached on the plurality of private clients is invalidated, and the second shard file metadata includes the data amounts of the plurality of shards included in the second shard file server.

[0035] Exemplarily, the master file server is further configured to send identifier setting information to the first shard file server; the first shard file server is further configured to set the second file data amount owner identifier according to the identifier setting information, and update the first shard file data amount included in the first shard file server to a second shard file data amount, where the second shard file data amount is determined according to the first shard identifier and the data amounts of the plurality of shards included in the file; the first shard file server is further configured to send second cache invalidation information to the plurality of private clients through the distributed lock server, so that the second shard file metadata corresponding to the first shard file server and cached on the plurality of private clients is invalidated, and the second shard file metadata includes the data amounts of the plurality of shards included in the first shard file server.

[0036] Exemplarily, the at least one private client in the system further includes a second private client; the second private client is configured to determine a second shard identifier of third data to be written, the second shard identifier being obtained based on a data offset address of the third data and a length of the third data; and further configured to send a second shard creation message to the master file server according to the cached third data layout table and the second shard identifier, the third data layout table including a third maximum shard identifier and a first bit, the third maximum shard identifier being greater than or equal to the second shard identifier, and a bit value of the first bit indicating that data included in the shard corresponding to the second shard identifier is invalid; the master file server is further configured to send data clearing information to a third shard file server to which the second shard belongs according to the second shard creation information and a stored fourth data layout table, where the fourth data layout table includes a fourth maximum shard identifier and a second bit, the fourth maximum shard identifier being greater than or equal to the second shard identifier, and a bit value of the second bit indicating that data included in the shard corresponding to the second shard identifier is invalid; the third shard file server is configured to clear the data included in the second shard in response to the data clearing information, and send a data clearing response to the master file server.

[0037] Exemplarily, the at least one private client in the system further comprises a third private client; the third private client is configured to determine a third shard identifier of fourth data to be written, the third shard identifier being based on a data offset address of the fourth data and a length of the fourth data; and further configured to send, to a fourth shard file server, a version number of a fifth data layout table and the third shard identifier, the fifth data layout table comprising a fifth maximum shard identifier and a third bit, the fifth maximum shard identifier being greater than or equal to the third shard identifier, and a bit value of the third bit indicating that a shard corresponding to the third shard identifier comprises valid data; the version number of the fifth data layout table being a version number of the fifth data layout table before a file truncation operation is performed on the file; and the master file server is further configured to send, to the fourth shard file server, a version number of a sixth data layout table, the version number of the sixth data layout table being a version number after the file truncation operation is performed on the file, and the version number of the sixth data layout table being greater than the version number of the fifth data layout table; and the fourth shard file server is configured to determine that the version number of the fifth data layout table and the version number of the sixth data layout table are different, and return a write data error to the third private client.

[0038] Exemplarily, the at least one private client in the system further comprises a fourth private client; the fourth private client is configured to send, to the master file server, a file truncation operation; the master file server is further configured to send, to one or more shard file servers corresponding to the file truncation operation, version number clearing information according to the file truncation operation; and further configured to update, according to the file truncation operation, a version number of a seventh data layout table in cache to a version number of a sixth data layout table, the version number of the seventh data layout table being a version number before the file truncation operation is performed on the file.

[0039] Exemplarily, the master file server is further configured to update, according to the file truncation operation, a third file data amount owner identifier included in the seventh data layout table to a fourth file data amount owner identifier included in the sixth data layout table, the third file data amount owner identifier being used to indicate a fifth shard file server to which a shard corresponding to a seventh maximum shard identifier belongs, and the fourth file data amount owner identifier being used to indicate a sixth shard file server to which a shard corresponding to a sixth maximum shard identifier belongs; and further configured to modify, according to the file truncation operation, a first bit value of one or more fourth bits included in the seventh data layout table to a second bit value, the fourth bits being used to indicate shards corresponding to the first file truncation operation, and the first bit value indicating that data included in the shards is valid, and the second bit value indicating that data included in the shards is invalid.

[0040] Exemplarily, the first private client is further configured to determine the first file attribute information according to the first attribute obtaining operation, the eighth data layout table cached, and the cached amounts of the shard file data included in the plurality of file servers, wherein the amount of the shard file data included in the first shard file server cached by the first private client is valid, and the first file attribute information includes at least one of the following: the amount of data of the target file, the creation time of the target file, and the modification time of the target file.

[0041] Exemplarily, the first private client is further configured to send data layout table obtaining information to the master file server and send data amount obtaining information to the first shard file server according to the second attribute obtaining operation, the ninth data layout table cached, and the cached amounts of the shard file data included in the plurality of file servers, wherein the amount of the shard file data included in the first shard file server cached by the first private client is invalid; and determine the second file attribute information according to the second data layout table sent by the master file server and the amount of the target shard file data sent by the first shard file server, wherein the second file attribute information includes at least one of the following: the amount of data of the target file, the creation time of the target file, and the modification time of the target file.

[0042] In a third aspect, the present application provides a computing device cluster, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the file semantic processing method provided in the first aspect.

[0043] In a fourth aspect, the present application provides a computer program product comprising instructions which, when executed by a computing device, cause the computing device cluster to implement the file semantic processing method provided in the first aspect.

[0044] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions which, when executed by a computing device, cause the computing device cluster to implement the file semantic processing method provided in the first aspect.

[0045] On the basis of the implementation manners of the aspects provided by the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0047] Figure 1 is a structure schematic diagram of a distributed file system provided by an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a data layout table provided by an embodiment of the present application;

[0049] Figure 3 is a data structure logical schematic diagram of a data layout table provided by an embodiment of the present application;

[0050] Figure 4 is a flowchart of a file processing method provided by an embodiment of the present application;

[0051] Figure 5 is a flowchart of another file processing method provided by an embodiment of the present application;

[0052] Figure 6 is a flowchart of a file truncation method provided by an embodiment of the present application;

[0053] Figure 7 is a flowchart of another file processing method provided by an embodiment of the present application;

[0054] Figure 8 is a flowchart of another file processing method provided by an embodiment of the present application;

[0055] Figure 9 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0056] Figure 10 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0057] Figure 11 is a structural schematic diagram of one or more computing devices connected through a network provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] Some names related to the present application will be defined first.

[0060] File index node (inode): a data structure in a file system, including file index node identification and corresponding file metadata, and the file metadata includes file size, file creation time, file modification time and other data.

[0061] The root inode (Inode) is the initial file inode when the metadata server to which the root inode belongs splits the file into multiple fragments. It is a structure that stores the overall file metadata information, including the overall file size, overall file creation time, and overall file modification time, and is used to manage the file.

[0062] A slice node is a file index node for multiple slices of a file. It is a structure that stores metadata information of the slices, including the slice size, creation time, modification time, and storage address of the slice in the object storage device. It is used to manage the multiple slices of a file and ensure data storage and access in the distributed file system.

[0063] A slice is a data unit obtained by splitting a file. The data units in a slice can be of fixed or variable size.

[0064] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a distributed file system provided in an embodiment of this application. The distributed file system includes a private client cluster, a metadata server cluster, and an object storage layer. The private client cluster includes at least one private client 110; the metadata server cluster includes multiple file servers, including a main file server 120 and multiple shard file servers 130; and the object storage layer includes multiple object storage devices (OSDs) 140.

[0065] Private client 110 has specific processing logic and permissions for performing file semantic operations such as reading, writing, truncate, and getting attributes, and connects to the main file server and the sharded file server via the network.

[0066] The main file server 120 and multiple sharded file servers 130 can be bare metal servers (BMS), virtual machines, or containers. A BMS refers to a general-purpose physical computing device, such as an ARM or x86 computing device; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.

[0067] The master file server 120 is configured to manage a master file inode 121 and a data map table 122, where the master file inode 121 includes metadata of a file (e.g., a file size, a file create time, a file modify time, a file access time, etc.). Figure 2 is a schematic diagram of a data map table provided by an embodiment of the present application, as shown in Figure 2 The data map table 122 includes, as shown, a slice size corresponding to an allocated slice, a slice file server number, a distribution strategy, a split threshold, a plurality of slice inodes, a plurality of slices, a file size owner identification, a slice truncate bitmap, and a data map table version number, etc.

[0068] The slice size is the data amount of each slice, which can be 1M or 128M, and the like, and the distribution strategy can be a strategy of performing slice allocation according to an order or a strategy of performing slice configuration according to affinity between the private client and the server, and the splitting threshold is used to indicate the maximum data amount that can be managed by the master file server, and in a case where the data written in the file is greater than the splitting threshold, the master file server needs to split the file, so that a plurality of file servers manage a plurality of slice files obtained by splitting; the slice file index node includes a parent file identifier of the slice file and a file identifier (FID) of the slice file, each slice file index node belongs to a file server; the slice includes a slice file index node identifier of the slice file to which the slice belongs; the file data amount owner identifier is used to indicate the slice file server to which the slice corresponding to the maximum slice identifier belongs, and can be an identifier of only one slice file index node included in the slice file server; the file data amount owner identifier is also used to determine the accurate file data amount, specifically, the slice data amount of a plurality of slices included in the slice file server with the file data amount owner identifier is obtained, and the file data amount can be calculated in combination with the slice size, the splitting threshold and the number of slice file servers; the slice truncation bitmap includes a plurality of bits, each bit is used to indicate whether the data included in the slice is valid, and the bit value changes in a case of file truncation operation; the data layout table version number can change in a case of file truncation, and is used to indicate the number of file truncation operations, and is used to ensure the consistency of file data between the private client, the master file server and the slice file server, and can be an integer greater than or equal to 0, and the specific limitation is not limited herein.

[0069] The process of splitting the target file of the private client by the master file server and creating the data layout table shown in FIG. 2 is described as follows: Figure 2

[0070] The private client determines the data amount of the first data according to the offset address and the data length of the first data to be written, and in a case where the sum of the data amount of the first data and the data amount of the second data written is greater than the splitting threshold, the file splitting information is sent to the master file server, wherein the sum of the data amount of the second data written is less than the splitting threshold. The master file server creates a slice file index node as shown in FIG. 2 according to the received file splitting information. Figure 2 ​The data layout table shown in the figure is determined according to the splitting strategy to determine the respective shard file servers to which the plurality of shard files belong. After determining the shard file servers, the master file server sends shard file creation information to the determined one or more shard file servers, so that each shard file server applies for a file identifier for storing data into the object storage device. After completing the shard file creation, the shard file server sends a shard file creation response to the master file server, so that the master file server stores the created data layout table into the object storage device, and then sends the data layout table to the private client, so that the private client performs data read / write, file truncation and other file semantic operations according to the data included in the data layout table.

[0071] Specifically, in the case of a splitting threshold of 1M, the private client determines that the sum of the data to be written 2.5M and the data already written 0.5M is 3M, and sends file splitting information to the master file server. The master file server creates a data layout table as shown in the figure, determines three shard file servers for managing shard files, and sends shard file creation information to the three shard file servers respectively. Then, the master file server determines that each shard size is 1M and the distribution strategy is to allocate according to the order, and allocates the first shard to the first shard file server, the second shard to the second shard file server, and the third shard to the third shard file server. In the case of continuing to write data by the private client, the fourth shard is allocated to the first shard file server, and then the same is applied to the subsequent shards, which will not be described here. Figure 2

[0072] In one possible implementation, Figure 2 This is only one possible implementation provided by the present application. The data layout table can also include more types and quantities of data, and can be represented in other forms, which are not limited by the present application.

[0073] It should be understood that Figure 2 The data layout table shown in the figure includes all types of data that can be identified and processed by the private client 110 to ensure the normal completion of file semantic operations such as data read / write, file truncation (truncate), attribute acquisition (getattr), etc.

[0074] ​The sharded file server 130 is used to manage sharded files. Each sharded file includes multiple shards allocated by the main file server. Specifically, the sharded file server is used to manage sharded file inodes 131. The sharded file inode 131 includes sharded file metadata (e.g., the amount of sharded file data in the sharded file server, the creation time, modification time, access time, etc. of the sharded file) and sharded data layout information. The data layout information is the storage address of the data corresponding to each shard in the object storage device. The sharded file server 130 is used to persist the sharded data, sharded file metadata, and sharded data layout information to the object storage device (OSD) 140.

[0075] In one possible implementation, the first slice file server, which includes the first slice file, and the master file server can be the same MDS. In this case, the first slice file inode (slice inode1), which includes the metadata of the first slice file and the data layout information of the related slices, can be managed by the master file server, just like the file inode (root inode).

[0076] In another possible implementation, the first fragment file server and the main file server can also be two different MDSs, which are not specifically limited in this application.

[0077] Figure 3 This is a logical diagram of the data structure of a data layout table provided in an embodiment of this application, such as... Figure 3 As shown, this embodiment of the application uses a splitting threshold of 2M, a fragment size of 1M, a number of fragment file servers of 4, and a distribution strategy of sequential allocation as an example for illustration. If the private client determines that the sum of the data to be written and the data already written is greater than 2MB, the master file server sends fragment file creation information to the four fragment file servers based on the file splitting information, and splits the file into multiple fragments of 1MB each. The first fragment (fragment 0) is assigned to the first fragment file server for management. At this time, the data volume of the fragment file in the first fragment file server has not reached the splitting threshold, so the second fragment (fragment 1) is assigned to the first fragment file server. At this time, the data volume of the fragment file in the first fragment file server reaches the splitting threshold. According to the sequential allocation distribution strategy, the master file server assigns the third fragment (fragment 2) to the second fragment file server, the fourth fragment (fragment 3) to the third fragment file server, and the fifth fragment (fragment 4) to the fourth fragment file server. Since there are four fragment file servers, the master file server assigns the sixth fragment (fragment 5) to the first fragment file server, and so on. This process will not be elaborated further here.

[0078] As shown in Figure 3 , three slices (slice 0, slice 1 and slice 5) in the first slice file server constitute a slice file, the interval (0, 1M) in the file corresponds to the interval (0, 1M) in the slice file, the interval (1, 2M) in the file corresponds to the interval (1, 2M) in the slice file, the interval (5, 6M) in the file corresponds to the interval (2, 3M) in the slice file, according to the interval (2, 3M) in the last slice file, it can be determined that the slice file data amount in the first slice file server is 3M, similarly, the corresponding relationship between the slice file in the second slice file server, the third slice file server and the fourth slice file server and the file is similar to the above process, it can be determined that the slice file data amount included in the second slice file server, the third slice file server and the fourth slice file server respectively is 2M, which will not be described in detail here.

[0079] According to the corresponding relationship between the data interval of the file in the slice file server and the data interval of the slice file, the private client can determine the slice corresponding to the data under the condition of determining the offset address and length of the data, for example, in the case that the offset address of the data is 1M and the length is 1M, the slice corresponding to the data is slice 1, in the case that the offset address of the data is 900K and the length is 200K, according to the slice size of 1M, it is determined that the data range crosses two slices, corresponding to slice 0 and slice 1.

[0080] In Figure 3 , the maximum slice identifier is 8, slice 8 is the last slice of the file, the fourth slice file server to which slice 8 belongs includes the file data amount owner identifier, wherein the file data amount owner identifier can be the identifier of the slice file index node included in the fourth slice file server (slice inode4). According to the slice data amount 2M of the plurality of slices included in the fourth slice file server, the split threshold 2M included in the data layout table, the slice size 1M and the slice file server quantity 4, it can be calculated that the file data amount is 9M. Compared with the method of directly obtaining the file data amount from the master file server, the method of determining the file data amount according to the file data amount owner identifier, the split threshold, the slice size and the slice file server quantity can obtain more accurate file data amount.

[0081] In the above process, the master file index node and the data layout table managed by the master file server, and the slice file index node managed by the slice file server can be cached in the private client, the private client performs various file semantic operations according to the cached data, for example, data read operation, data write operation, file attribute information acquisition operation, file truncation operation, etc., which will not be limited herein.

[0082] The object storage device 140 is configured to store a data layout table, data of a file, metadata of the file, metadata of a shard file, and the like, which are not limited in the present application.

[0083] The distributed file system further comprises a distributed lock manager (DLM) server 150, which can be deployed in the master file server or the shard file server, or independently deployed in the distributed file system, and is configured to manage validity of metadata of the file included in the private client cache master file inode, metadata of the shard file included in the shard file inode, and data layout information of the shard.

[0084] It should be understood that Figure 1 The distributed file system can further comprise more types and quantities of devices, which are not limited in the present application.

[0085] As Figure 4 shown, Figure 4 is a flowchart of a file processing method provided by an embodiment of the present application, which is applied to Figure 1 the distributed file system shown in the figure, and in a scenario where a first shard corresponding to first data to be written by a first private client does not exist, the method comprises the following steps.

[0086] S401: The first private client determines a first shard identifier of the first data to be written.

[0087] The first private client determines the first shard identifier of the first shard corresponding to the first write operation according to the offset address and the length of the first data to be written, and according to the shard size in the cached data layout table.

[0088] Specifically, when the first private client determines that the offset address of the data to be written is 10M and the length is 1M, and the shard size is 1M according to the cached data layout table, the first private client determines that the first shard identifier of the first shard corresponding to the first write operation is 10.

[0089] In a possible implementation, the first private client determines a plurality of shards corresponding to the data to be written according to the offset address and the length of the data to be written, and the shard size in the cached data layout table, and at this time, the first private client takes the largest shard identifier in the plurality of shard identifiers corresponding to the plurality of shards respectively as the first shard identifier.

[0090] For example, in a case where the first private client determines that the offset address corresponding to the data to be written is 7M, the length is 3M, and the size of the shard is 1M according to the cached data layout table, it is determined that the data to be written corresponds to three shards, and the maximum shard identifier 9 in the shard identifiers corresponding to the three shards is determined as the first shard identifier.

[0091] S402: The first private client sends the first shard creation information to the master file server according to the cached first data table and the first shard identifier.

[0092] The first private client determines that the first maximum shard identifier included in the first data layout table is smaller than the first shard identifier according to the first shard identifier and the cached first data layout table, determines that the first shard does not exist, and sends the first shard creation information to the master file server.

[0093] Specifically, in a case where the first shard identifier is determined to be 10 and the cached first data layout table is as shown in Figure 3 The first maximum shard identifier is 8, so it can be determined that the first shard identifier is greater than the first maximum shard identifier included in the cached first data layout table, and the first private client determines that the first shard does not exist and sends the first shard creation information to the master file server.

[0094] In a possible implementation, in a case where the data to be written corresponds to multiple shards, the maximum shard identifier in the shard identifiers corresponding to the multiple shards is compared with the first maximum shard identifier, and in a case where it is greater than the first maximum shard identifier, it is determined that the first shard does not exist and the first shard creation information is sent to the master file server.

[0095] Specifically, in a case where the maximum shard identifier of the three shards corresponding to the data to be written is 9 and the cached first data layout table is as shown in Figure 3 The first maximum shard identifier is 8, so it can be determined that the first shard identifier is greater than the first maximum shard identifier, the first private client determines that two shards with shard identifiers 7 and 8 exist among the three shards corresponding to the first write operation, and the shard with the shard identifier 9 does not exist, and sends the first shard creation information to the master file server.

[0096] S403: The master file server determines whether the first shard exists and whether the included data is valid according to the received first shard creation information and the stored first data layout table.

[0097] The master file server determines whether the first shard exists according to the first maximum shard identifier in the first data layout table after receiving the first shard creation information.

[0098] The master file server, in a case where it is determined that the first shard does not exist, performs step S404.

[0099] The master file server, in a case where it is determined that the first shard exists, continues to determine whether the data included in the first shard is valid according to the plurality of bits included in the first data layout table. In a possible implementation, in a case where the value of the bit indicating the first shard is 0, it is determined that the data included in the first shard is valid, and in a case where the value of the bit indicating the first shard is 1, it is determined that the data included in the first shard is invalid, where the bit value indicating that the data included in the shard is valid or invalid is only one possible example, which is not limited by the present application.

[0100] The master file server, in a case where it is determined that the first shard exists and the data included in the first shard is valid, performs step S406.

[0101] The master file server, in a case where it is determined that the first shard exists but the data included in the first shard is invalid, performs step S407.

[0102] S404: The master file server creates the first shard according to the first shard creation information and the stored second data layout table.

[0103] The master file server creates the first shard according to the first shard creation information, modifies the first maximum shard identifier to the first shard identifier, modifies the first file data amount owner identifier to the second file data amount owner identifier, and modifies the first shard number to the second shard number, thereby realizing the modification of the second data layout table to the second file metadata, where the second file data amount owner identifier is used to indicate the first shard file server to which the first shard belongs, the second shard number is greater than the first shard number, and the second shard number is determined according to the first shard identifier.

[0104] Specifically, the master file server, in a case where it is determined according to the first shard creation information that the first shard identifier of the first shard is 10 and the second data layout table is as shown in Figure 3 , modifies the first maximum shard identifier (8) to the first shard identifier (10), modifies the first file data amount owner identifier (sliceinode4) to the second file data amount owner identifier (sliceinode2), and modifies the first shard number (9) to the second shard number (11), thereby realizing the update of the second data layout table.

[0105] The master file server sends the file data volume owner identification clearing information to the second shard file server indicated by the first file data volume owner identification in the process of modifying the first data layout table, so that the second shard file server clears the file data volume owner identification and sends the file data volume owner identification clearing response to the master file server. Moreover, the master file server sends the file data volume owner identification setting information to the first shard file server indicated by the first file data volume owner identification, so that the first shard file server sets the file data volume owner identification and the total data volume of the plurality of shards including the first shard in the first shard file server, and sends the file data volume owner identification setting response information to the master file server.

[0106] The above process determines the new file data volume owner, and according to the shard data volume of all shards included in the first shard file server with the file data volume owner identification set, and the unchanged shard size, split threshold and shard file server number, the file data volume after the first shard is created can be determined, rather than being determined according to the shard data volume of the second shard file server indicated by the first file data volume owner identification before the shard is created, so that the accurate determination of the file data volume can be ensured.

[0107] After the file data volume owner identification clearing or setting operation is performed by the first shard file server and the second shard file server, the metadata of the shard file updated by each of the first shard file server and the second shard file server is stored in the storage device. Then, the second shard file server sends the first cache invalidation information to the plurality of private clients through the distributed lock server, so that the first shard file metadata corresponding to the second shard file server cached on the plurality of private clients is invalidated, and the storage addresses of the plurality of shards included in the first shard file server are invalidated. The first shard file metadata includes the storage addresses of the plurality of shards included in the first shard file server in the storage device, the total data volume of the plurality of shards, the creation time, the modification time, the access time and the like of the shard file. Similarly, the first shard file server sends the second cache invalidation information to the plurality of private clients through the distributed lock server, so that the second shard file metadata corresponding to the first shard file server cached on the plurality of private clients is invalidated, and the storage addresses of the plurality of shards included in the first shard file server are invalidated.

[0108] In the above process, the first shard file server and the second server send the cache invalidation information to the plurality of private clients, which can prevent the private clients from performing the property acquisition operation and other file semantic operations according to the cached shard file metadata and the storage addresses of the shards during the shard creation process, so as to cause the inaccurate data of the calculated file size, or the problem that the file semantic operation cannot be normally completed.

[0109] S405: The master file server sends the updated second data layout table to the first private client.

[0110] After receiving the file data volume owner identifier clearing response sent by the second shard file server and the file data volume owner identifier setting response sent by the first shard file server, the master file server sends the updated second data layout table to the first private client.

[0111] Through the above process, the private client determines whether to send the shard creation information according to the maximum shard identifier of the cached data layout table, and the master file server updates the target file modification time, the data volume of the target file and other data in the process of creating the shard, so as to ensure the accuracy of the data layout table. The master file server sends the updated data layout table to the private client, so that the private client can accurately update the data layout table to determine the file attribute information in the case of subsequent attribute acquisition operation, so as to determine the accurate file size after the first shard is created.

[0112] S406: The master file server sends the stored second data layout table to the first private client.

[0113] The master file server sends the stored first data layout table to the first private client in the case that the first shard exists and the data included in the first shard is valid according to the first shard creation information and the stored first data layout table.

[0114] S407: The master file server modifies the second data layout table and sends the modified second data layout table to the first private client.

[0115] The master file server sends data clearing information to the first shard file server to which the first shard belongs in the case that the first shard exists but the data included in the first shard is invalid according to the first shard creation information and the stored second data layout table, so that the first shard file server clears the dirty data included in the first shard and sends a data clearing response to the master file server.

[0116] The master file server determines the first bit corresponding to the first shard according to the first data layout table, and modifies the second bit value corresponding to the first bit to the first bit value, wherein the first bit value indicates that the data included in the shard is valid, and the second bit value indicates that the data included in the shard is invalid. The application does not make specific limitation on the values of the first bit value and the second bit value.

[0117] The master file server sends the second data layout table with the modified bit value to the first private client after receiving the data clearing response sent by the second shard file server to which the first shard belongs.

[0118] S408: The first private client updates the cached data layout table, and sends the first shard identifier and the version number of the updated cached data layout table to the first shard file server to which the first shard belongs.

[0119] The main file server performs steps S405, S406 and S407, and can update the data layout table cached by the first private client. The updated cached data layout table can be the updated second data layout table, the stored second data layout table, or the second data layout table with the modified bit value.

[0120] In a possible implementation, the first private client sends the first write operation and the version number included in the updated cached data layout table to the first shard file server to which the first shard belongs, to write data into the storage device, in a case where the first shard corresponding to the first write operation exists according to the updated cached data layout table, and the data included in the first shard is valid.

[0121] In another possible implementation, the first private client can still perform steps S402-S408 in a case where the first shard corresponding to the first write operation exists according to the first write operation and the cached data layout table, but the data included in the first shard is invalid. Specifically, in a case where the first shard identifier corresponding to the first shard is less than or equal to the maximum shard identifier included in the cached data layout table, the first private client determines that the first shard exists, and then determines, according to the cached data layout table, a bit in the plurality of bits indicating the first shard. In a case where the bit value of the bit corresponding to the bit is a bit value indicating that the data included in the shard is invalid, the first private client determines that the data included in the first shard is invalid.

[0122] In another possible implementation, in a case where the first shard corresponding to the first read operation does not exist according to the cached data layout table and the first read operation, or the first shard exists but the data included in the first shard is invalid, the first private client and the main file server do not perform the steps shown in FIG. 4, but stop sending the first read operation and generate a read data out-of-bound alarm information. Figure 4

[0123] In summary, the main file server can create a shard in a case where the shard corresponding to the data write operation sent by the private client does not exist. The creation of the shard involves modification of the data layout table, which can affect the data such as the file data amount owner identifier included in the data layout table. The main file server updates the data layout table cached by the private client in different scenarios, to ensure the accuracy of the data layout table, so that the performance of the private client in performing operations such as data read-write operation and attribute acquisition operation according to the updated data layout table. ​

[0124] As Figure 5 shown, Figure 5 is a flowchart of another file processing method provided by the embodiments of the present application. The method is applied to a distributed file system as shown in Figure 1 , and in a scenario where a first shard corresponding to first data to be written by a first private client exists and the data included in the first shard is valid, the method comprises the following steps.

[0125] S501: The first private client determines the first shard identifier of the first data to be written, and determines that the first shard exists and is valid according to the cached first data layout table.

[0126] The first private client determines the first shard identifier of the first shard corresponding to the first data to be written according to the offset address and length of the first data to be written, and according to the shard size in the cached first data layout table. The first private client compares the maximum shard identifier included in the cached first data layout table with the first shard identifier. In the case where the first shard identifier is less than or equal to the maximum shard identifier included in the first data layout table, the first private client determines that the first shard exists. The first private client determines the bit indicating the first shard among the plurality of bits according to the first data layout table. In the case where the bit value of the bit corresponding to the first shard is a bit value indicating that the data included in the shard is valid, the first private client determines that the data included in the first shard is valid.

[0127] Specifically, in the case where the first private client determines that the offset address of the data to be written is 4M, the length is 1M, and the shard size is 1M according to the cached first data layout table, the first shard identifier of the first shard corresponding to the first data to be written is determined to be 4. In the case where the cached first data layout table is as shown in Figure 3 , it is determined that the maximum shard identifier is 8, so that it can be determined that the first shard identifier is less than the maximum shard identifier included in the first data layout table, and the first private client determines that the first shard exists. The first private client determines the bit value of the bit indicating the first shard among the plurality of bits in the first data layout table to be 0, and determines that the data included in the first shard is valid.

[0128] S502: The first private client sends the first shard identifier and the version number of the first data layout table to the first shard file server.

[0129] S503: The first shard file server determines whether the version number of the first data layout table is the same as the version number obtained from the master file server.

[0130] In the case where the first private client determines that the first version number is the same as the version number obtained from the master file server, the first shard file server performs step S504;

[0131] The first shard file server executes step S505 in a case where the first version number is determined to be different from the version number obtained from the master file server.

[0132] S504: The first shard file server writes the first data to be written into the object storage device.

[0133] In a case where the first version number is determined to be the same as the version number obtained from the master file server, the first shard file server writes the first data to be written into the storage device, then updates the shard file metadata, caches the storage address of the first data in the object storage device, and stores the shard file metadata and the storage address into the object storage device.

[0134] In the above process, the first version number and the version number obtained from the master file server are used to determine whether the data layout tables cached by the first shard file server and the first private client are the same, and to determine whether there is a concurrent file truncation operation in the execution process of the write operation. Only in the case where the version numbers are the same, the write operation is executed, which can ensure the consistency of the data of the first private client in the case where other private clients perform file truncation operations and other file semantic operations.

[0135] In the above process, in a case where the data length corresponding to the first data to be written is short, for example, only one shard is involved, the first shard file server sends the first data to be written to the first shard file server, so that the first shard file server executes the steps. However, in a case where the data length corresponding to the first data to be written is long, for example, multiple shards are involved, the first private client sends the first data to be written to the object storage device, and after the object storage device completes the data write operation, writes the updated shard file metadata included in the shard file server and the storage address of the data in the object storage device into the corresponding shard file server.

[0136] S505: The first shard file server returns a write data error.

[0137] In a case where the first version number is determined to be different from the version number cached locally, the first shard file server stops executing the first data to be written, and returns a write data error.

[0138] In a possible implementation, in a scenario where the first private client determines, according to the cached data layout table and the first read operation, that the first shard corresponding to the first read operation exists, and the first shard includes valid data, the first shard file server can execute steps S501 and S502.

[0139] In the above process, when the data corresponding to the first read operation is short, for example, only one shard is involved, the first shard file server reads the data and sends the data to the private client upon receiving the first read operation sent by the first private client. However, when the data corresponding to the first read operation is long, for example, multiple shards are involved, the first shard file server sends the storage address of the data in the object storage device to the private client upon receiving the first read operation sent by the first private client, so that the private client reads the data directly from the object storage device according to the received storage address of the data.

[0140] In the above process, the first shard file server determines that the version numbers are different because the first private client concurrently sends the first data to be written to the first shard file server in the process in which the master file server performs a file truncate operation on the shards in the first shard file server. The file truncate operation is an operation of setting the file size. For example, when the shard size and the split threshold are both 1M, the file truncate operation can set the file data amount from 5M to 1M. This operation is to invalidate the data included in the second shard to the fifth shard respectively, and change the data into dirty data. Only the 1M data included in the first shard is still valid. The file truncate operation can also set the file data amount from 1M to 5M. This operation is to clean the dirty data included in the second shard to the fifth shard respectively, so that the corresponding 4M data is valid.

[0141] Because the data layout table includes multiple bits indicating whether the data in the shard is valid, the master file server performs a file truncate operation, modifies the bit values of the multiple bits in the data layout table indicating the second shard to the fifth shard file, changes the bit value indicating that the data included in the shard is valid to the bit value indicating that the data included in the shard is invalid, and changes the file data amount owner identifier, thereby updating the data layout table and updating the version number of the data layout table. The first private client sends the first version number of the data layout table before the file truncate operation and the first data to be written to the first shard file server in the process of the file truncate operation. The first shard file server obtains the version number of the updated data layout table after the file truncate operation from the master file server, and determines that the version numbers are different.

[0142] The specific process of the file truncate operation causing the version numbers to be different is shown in Figure 6 The flowchart of a file truncate method provided by an embodiment of the present application is shown in Figure 6 The method is applied to the distributed file system shown in Figure 1 The method includes the following multiple steps.

[0143] S601: The second private client sends a file truncation operation to the master file server.

[0144] S602: The master file server sends version number clearing information to one or more shard file servers corresponding to the file truncation operation according to the file truncation operation.

[0145] The master file server receives the file truncation operation sent by the second private client, and determines one or more shards involved in the file truncation operation and one or more shard file servers to which each shard in the one or more shards belongs according to the file truncation operation and the stored second data layout table, wherein the second data layout table includes shard size, split threshold and shard file server quantity. The master file server sends version number clearing information to a plurality of shard file servers to which a plurality of shards respectively belong, wherein the version number clearing information is used to instruct the shard file servers to invalidate the version number of the data layout table before the file truncation operation in the local cache.

[0146] For example, the file truncation operation sent by the second private client is an operation of changing the file data volume from 9M to 4M, and in the case of the second data layout table being the data layout table shown in Figure 3 The master file server determines that the file truncation operation corresponds to the fifth shard to the ninth shard, and determines a plurality of shard file servers to which the fifth shard to the ninth shard belong according to the shard size, the split threshold and the shard file server quantity included in the first data layout table, wherein the plurality of shard file servers include the first shard file server, the second shard file server, the third shard file server and the fourth shard file server. The master file server sends version number clearing information to the first shard file server, the second shard file server, the third shard file server and the fourth shard file server, so as to invalidate the version number of the second data layout table cached in the local cache.

[0147] In the above process, the master file server sends version number clearing information to a plurality of shard file servers, invalidates the version number of the data layout table before the file truncation operation cached in the shard file server, so that the shard file server receives the version number of the data layout table after the file truncation operation sent by the master file server in the case of receiving the file semantic operation sent by the private client concurrently and the version number of the data layout table before the file truncation operation, and judges the version number, thereby avoiding the adverse effects of directly executing the file semantic operation on data consistency.

[0148] S603: The master file server updates the second data layout table to a third data layout table according to the file truncation operation.

[0149] The master file server determines one or more shards corresponding to the file truncation operation and a third file data quantity owner identifier according to the file truncation operation and the second data layout table, where the third file data quantity owner identifier is used to indicate a third shard file server to which a shard corresponding to a maximum shard identifier in the file after the file truncation operation belongs.

[0150] The master file server modifies the second file data quantity owner identifier in the second data layout table to the third file data quantity owner identifier, and modifies a bit indicating the one or more shards corresponding to the file truncation operation from a first bit value to a second bit value, where the second file data quantity owner identifier is used to indicate a second shard file server to which a shard corresponding to a maximum shard identifier in the file before the file truncation operation belongs, and the first bit value indicates that the shard includes valid data, and the second bit value indicates that the shard includes invalid data.

[0151] In a specific implementation, in the case of the data layout table shown in FIG. 4, the master file server determines that the file truncation operation sent by the second private client corresponds to the fifth shard to the ninth shard, the maximum shard identifier in the file after the file truncation operation is 3, the shard 3 belongs to the third shard file server, and the file data quantity owner identifier after the file truncation operation is an identifier of a third shard file index node included in the third shard file server (sliceinode3). Figure 3 The master file server modifies the second file data quantity owner identifier (sliceinode4) in the second data table to the identifier of the third shard file index node (sliceinode3), and modifies the first bit value (0) of the five bits in the second data table indicating the fifth shard to the ninth shard to the second bit value (1).

[0152] In addition to modifying the file data quantity owner identifier and the plurality of bit values in the data layout table, the master file server also modifies the second version number of the second data layout table to a third version number, so as to modify the second data layout table to a third data layout table, and store the third data layout table in the object storage device. In the above process, the modified third version number can be sent to the private client and the shard file server, and is used to indicate that the file truncation operation is completed normally, and to ensure the consistency of data in the scenario of concurrent execution of data read and write operations by the private client during the file truncation operation, and the specific process is as shown in FIG. 5, which will not be described herein again. Figure 5

[0153] ​The master file server sends file data quantity owner identifier clearing information to the second shard file server indicated by the second file data quantity owner identifier to make the second shard file server clear the second file data quantity owner identifier after determining the second file data quantity owner identifier before the file truncation operation and the third file data quantity owner identifier after the file truncation operation, and sends a file data quantity owner identifier clearing response to the master file server; the master file server also sends file data quantity owner identifier setting information to the third shard file server indicated by the third file data quantity owner identifier to make the third shard file server set the third file data quantity owner identifier, determine the shard data quantity of the plurality of shards included by the third shard file server after the file truncation operation, and send a file data quantity owner identifier setting response to the master file server. The third shard file server sets the third file data quantity owner identifier, and in the case of receiving file size obtaining information sent by the private client, sends the shard data quantity of the plurality of shards cached locally after the file truncation operation to the private client, so that the private client determines the accurate file data quantity after the file truncation operation according to the shard data quantity and the shard size, the split threshold, etc., avoiding the private client sending the file size obtaining information to the second shard file server according to the second file data quantity owner identifier before the file truncation operation, which causes the error calculation of the file data quantity.

[0154] It should be understood that in the above process, the second shard file server and the third shard file server, like in the shard creation process, send their respective cache invalidation information to the plurality of private clients, so that the second shard file server and the third shard file server each corresponding shard file metadata cached on the plurality of private clients is invalidated, and the storage addresses of the plurality of shards each included are invalidated, thereby ensuring data consistency.

[0155] In a specific embodiment, in the case of the first data layout table being the data layout table shown in Figure 3 The master file server determines that the file truncation operation corresponding to the fifth shard to the ninth shard sent by the second private client, sends file data quantity owner identifier clearing information to the fourth shard file server, and sends file data quantity owner identifier setting information to the third shard file server, so that the third shard file server sets the file data quantity owner identifier as the identifier of the third shard file inode (sliceinode3), and determines that the file data quantity included after the file truncation operation is 1M, and according to the shard data quantity included by the third shard file server, the shard size, the split threshold, and the number of shard file servers, the file data quantity after the file truncation operation can be determined as 4M.

[0156] S604: The master file server sends the third data layout table to the second private client.

[0157] The master server sends the third data layout table to the second private client, enabling the second private client to perform various file semantic operations based on the updated third data layout table.

[0158] Currently, the master file server performs file truncation operations without managing the data layout table, which includes bits indicating fragments, file data size owner identifiers, and version numbers. Even if the file truncation operation for some fragments fails to complete correctly, the master file server may still determine that the file truncation operation was completed normally, adversely affecting file integrity and availability. Furthermore, recovering from such abnormal file truncation operations is difficult, and it is impossible to accurately determine which fragments require re-execution of the file truncation operation. However, the file truncation operation provided in this application embodiment allows the master file server to accurately determine whether the file truncation operation for one or more fragments has been completed normally by using multiple bits included in the data layout table, the file data size owner identifier, and the version number. It can accurately and efficiently identify the fragments whose file truncation operation failed to complete normally through the bit value of each bit.

[0159] Furthermore, currently, without managing a data layout table that includes bits indicating fragment size, file data size owner identifier, and version number, the main file server needs to limit the number of fragment file servers to ensure the performance of file truncation operations and avoid involving more fragment file servers, thus affecting execution efficiency. However, limiting the number of fragment file servers impacts the performance of concurrent read / write operations by private clients. In contrast, the embodiments of this application can determine the completion status of file truncation operations based on bits indicating fragment size, file data size owner identifier, and version number. Therefore, it is not necessary to limit the number of fragment file servers, and the performance of concurrent read / write operations by private clients can be guaranteed while ensuring the performance of file truncation operations.

[0160] like Figure 7 As shown, Figure 7 This is a flowchart of another file processing method provided in an embodiment of this application, which is applied to... Figure 1 In the distributed file system shown, the first private client executes on the main file server. Figure 6 In the scenario of concurrent write operations during the file truncation operation shown, this method includes the following steps:

[0161] S701: The first private client sends the first data to be written and the version number included in the first data layout table to the first fragment file server.

[0162] The first private client determines the existence of the first fragment based on the fragment size, maximum fragment identifier, and the offset address and length corresponding to the first data to be written, as included in the cached first data layout table. The specific determination process is as follows: Figure 4 As described in step S401, if the first fragment identifier of the first fragment is less than or equal to the largest fragment identifier included in the cached data layout table, the first private client determines that the first fragment exists. The first private client determines whether the data included in the first fragment is valid based on multiple bits included in the cached first data layout table. Specifically, the first private client determines the bit value of the bit indicating the first fragment in the first data layout table; if the bit value indicates that the data included in the fragment is valid, the first private client determines that the data included in the first fragment is valid.

[0163] When the first private client determines that the first data to be written corresponds to the first fragment exists and the data included in the first fragment is valid, based on the first data layout table in the cache, the first private client determines the first fragment file server to which the first fragment belongs based on the fragment size, split threshold and number of fragment file servers included in the first data layout table, and sends the first data to be written and the version number included in the first data layout table to the first fragment file server. The first version number is the version number of the first data layout table before the main file server performs the file truncation operation.

[0164] In one possible implementation, the data offset address corresponding to the first data to be written is 7M, and the length is 1M. The first data layout table is as follows: Figure 3 In the scenario described, the first private client determines that the first fragment corresponding to the first data to be written exists, and the first fragment identifier is 7. Since the first fragment identifier is less than the maximum fragment identifier 8 included in the first data layout table, the first private client determines that the first fragment exists. The first private client determines that the value of the bit indicating the first fragment in the first data layout table is 0, thus determining that the data included in the first fragment is valid.

[0165] The first private client determines the ownership of the first fragment based on the fragment size of 1MB, the split threshold of 2MB, and the number of fragment file servers of 4, as included in the first data layout table. Figure 3 In the third shard file server, the first private client sends the first data to be written and the first version number of the first data layout table to the third shard file server.

[0166] S702: The master file server sends the version number of the third data layout table to the first fragment file server.

[0167] Because the main file server is executing Figure 6In the process of the file truncation operation shown, the version number clearing information is first sent to the plurality of shard file servers involved in the file truncation operation in step S601. In the case where the plurality of shard file servers include the first shard file server, the first shard file server invalidates the version number of the cached data layout table. In the case where the first shard file server receives the first data to be written and the first version number and needs to perform version number comparison, the first shard file server sends version number acquisition information to the master file server, so that the master file server sends the version number of the third data layout table to the first shard file server. The version number of the third data layout table is the version number of the data layout table after the master file server performs the file truncation operation.

[0168] S703: The first shard file server determines that the version number of the first data layout table and the version number of the third data layout table are different, and returns a write data error to the first private client.

[0169] After receiving the version number of the third data layout table sent by the master file server, the first shard file server compares the version number of the first data layout table and the version number of the third data layout table. Since the version number of the third data layout table is the version number of the updated data layout table after the master file server performs the file truncation operation, and the version number of the first data layout table is the version number of the data layout table before the master file server performs the file truncation operation, the first shard file server determines that the version number of the first data layout table and the version number of the third data layout table are different, and returns a write data error to the first private client.

[0170] S704: The first shard file server sends data layout table update information to the first private client.

[0171] Compared with the method of concurrently performing data read and write operations by the current private client and the shard file server without including the version number of the data layout table, in the above process, the first shard file server determines that the file data corresponding to the first data layout table and the second data layout table are inconsistent in the case where the second version number is different from the first version number, and does not perform the method of writing the first data to be written. The occurrence of the situation that the private client reads the data that has not been cleared, or writes the data to the shard where the dirty data has not been cleared, etc. can be avoided, so that the consistency of the concurrent read and write data of the private client during the file truncation operation of the master file server can be ensured.

[0172] The first shard file server also sends data layout table update information to the first private client. The data layout table update information is used to instruct the first private client to obtain the third data layout table after the file truncation operation from the master file server, and perform the write operation of the first data according to the third data layout table.

[0173] S705: The first private client receives the third data layout table sent by the master file server according to the data layout table update information, and determines whether the first shard corresponding to the first data to be written exists and is valid according to the third data layout table.

[0174] The first private client acquires the third data layout table from the master file server according to the data layout table update information, and then determines whether the first shard corresponding to the first data to be written exists and is valid according to the third data layout table. The specific process of determining whether the first shard exists and is valid is shown in step S701, which will not be described here.

[0175] In a possible implementation, when the first private client determines, according to the third data layout table, that the first shard corresponding to the first write operation exists and the data included in the first shard is valid, the first private client sends the first write operation and the second version number to the first shard file server, so that the first shard file server executes the first write operation when determining that the received second version number is the same as the stored second version number. After executing the first write operation, the first shard file server updates the metadata of the corresponding shard file and the layout information of the shard data, and stores the above data in the object storage device, which is the same as step S504, thereby ensuring that the first private client writes data at a correct position or reads correct data, and improving the performance of the first write operation and the consistency of data.

[0176] In another possible implementation, when the first private client determines, according to the second data layout table, that the first shard corresponding to the first write operation exists but is invalid, the first private client sends shard creation information to the master file server, so that the master file server executes Figure 4 The specific process is shown in step S407, which will not be described here.

[0177] In a possible implementation, when the first write operation corresponds to multiple shards, the first private client splits the first write operation into multiple write operations, and sends each write operation and the first version number to the shard file server to which each shard corresponding to the write operation belongs, so that the shard file server executes Figure 7 The specific process is shown in step S407, which will not be described here.

[0178] In a possible implementation, when the first private client executes the first read operation, the first private client executes Figure 7 The specific process is shown in step S701, which will not be described here. When the first private client determines, according to the first data layout table, that the first shard corresponding to the first read operation exists and the data included in the first shard is valid, the first private client sends the first read operation and the first version number to the shard file server to which the first shard belongs, so that the shard file server executes Figure 7The steps S703 and S704 are shown. After that, the first private client executes Figure 7 The steps S705 are shown. In a case where it is determined that the first shard corresponding to the first read operation exists and the data included in the first shard is valid, the first read operation and the second version number are sent to the shard file server to which the first shard belongs; in a case where it is determined that the first shard corresponding to the first read operation exists but is invalid, the execution of the first read operation is stopped.

[0179] In summary, the shard file server determines whether to execute the data read-write operation sent by the first private client in the process of executing the file truncation operation on the master file server by checking whether the version number of the data layout table sent by the first private client is the same as the version number of the data layout table sent by the master file server, which can protect the consistency of the data and avoid the case where the first private client reads the dirty data involved in the file truncation operation or writes data to the invalid shard.

[0180] As shown in Figure 8 , Figure 8 is a flowchart of another file processing method provided by an embodiment of the present application. The method is applied to Figure 1 a distributed file system as shown, and in a scenario where a private client executes an attribute acquisition operation (getattr), the method includes the following steps.

[0181] S801: The private client determines whether the data included in the shard file server indicated by the file data volume owner identifier is valid according to the cached data layout table.

[0182] The data included in the shard file server is shard file metadata and data layout information, wherein the shard file metadata is the shard file data volume, the creation time, the modification time, and the access time of the shard file, and the shard file is composed of one or more shards included in the shard file server; and the data layout information is the storage address of the one or more shards included in the shard file server in the object storage device.

[0183] Since the shard file server sends cache invalidation information to the private client that caches the shard file metadata and the data layout information corresponding to the shard file server in a case where the file data volume owner identifier setting operation or the file data volume owner identifier clearing operation is executed, so as to invalidate the shard file metadata and the data layout information, the private client needs to first determine whether the data included in the shard file server indicated by the cached file data volume owner identifier is valid when executing the file attribute information acquisition operation.

[0184] In a case where the data included in the shard file server indicated by the cached file data volume owner identifier is valid, the private client executes step S802.

[0185] In the case where the data included in the shard file server indicated by the cached file data amount owner identifier is invalid, the private client performs step S803.

[0186] S802: The private client determines file attribute information according to the attribute acquisition operation, the cached data layout table, and the cached shard file data amount included in each of the plurality of file servers.

[0187] The file attribute information includes at least one of the file data amount and the file creation time, the file modification time. The file attribute information further includes more types and amounts of data, which are not limited in the present application.

[0188] In the case where the data included in the shard file server indicated by the cached file data amount owner identifier is valid, the private client can determine the file creation time, the modification time, and the file data amount according to the cached file metadata. However, since the file data amount in the file metadata is not accurate, the private client determines the shard file data amount included in the shard file server according to the data included in the shard file server indicated by the cached file data amount owner identifier. After determining the shard file data amount, the private client calculates the accurate file data amount according to the shard size, the split threshold, and the number of shard file servers in the cached data layout table.

[0189] Specifically, in the case where the shard size is 1M, the split threshold is 1M, the number of shard file servers is 4, the file data amount owner identifier is the first shard file inode identifier, and the shard data amount included in the first shard file server indicated by the first shard file inode identifier is 3M, the private client determines that the file data amount is 9M.

[0190] In the above process, in the case where the data included in the shard file server indicated by the cached file data amount owner identifier is valid, the private client determines that there is no concurrent file semantic operation such as file truncation operation or shard creation operation in the system. At this time, according to the cached shard data amount and the shard size, the split threshold, and the number of shard file servers in the data layout table, the accurate file data amount can be determined.

[0191] S803: The private client sends data layout table acquisition information to the master file server and sends data amount acquisition information to the first shard file server.

[0192] The private client sends data layout table obtaining information to the master file server and sends data amount obtaining information to the first file server indicated by the file data amount owner identifier included in the cached data layout table when the data included in the file server indicated by the file data amount owner identifier cached by the private client is invalid.

[0193] The first file server needs to determine whether the file data amount owner identifier is cached locally when receiving the data amount obtaining information. When the file data amount owner identifier is cached, the file server sends the file data amount of one or more shards included in the file server to the private client. When the file data amount owner identifier is not cached, the file server does not send the file data amount of one or more shards to the private client, but sends data layout table updating information to the private client to make the private client obtain the latest data layout table from the master file server, determine the file server indicated by the updated file data amount owner identifier according to the obtained latest data layout table, and send data obtaining information to the re-determined file server to obtain the file data amount sent by the file server indicated by the updated file data amount owner identifier.

[0194] In the above process, the private client determines that there is a concurrent file semantic operation such as file truncation operation or shard creation operation in the system when the data included in the file server indicated by the file data amount owner identifier cached by the private client is invalid. The private client sends data obtaining information to the file server indicated by the file data amount owner identifier cached by the private client. Since the file semantic operation such as file truncation operation or shard creation operation can modify the file data amount owner identifier, the file server needs to determine whether the file data amount owner identifier is set in the file server after receiving the data obtaining information. Only when the file data amount owner identifier is set in the file server, the file data amount is sent to the private client, thereby ensuring the accuracy of calculating the file data amount.

[0195] Compared with the current file attribute information obtaining operation, the private client needs to send data obtaining information to the master file server and each of the slice file servers including the slices, and determine the file metadata and the accurate file data amount by adding the slice data amounts sent by the multiple slice file servers respectively. In the case that the private client allocates the slices to more slice file servers, the performance of the private client in concurrent data read-write operation is improved, but the file attribute information obtaining operation needs to send file data amount obtaining information to more slice file servers, and the calculation resource consumption is large and the performance is poor. In the file semantic processing method provided in the above embodiment of the present application, the private client can determine the file data amount according to the cached file metadata, or according to the file metadata sent by the master file server and the slice file server with the file data amount owner identifier, the slice size, the split threshold, the number of slice file servers and the slice data amount corresponding to the slice file server, and does not need to send file data amount obtaining information to multiple slice file servers, so that the calculation resource consumption can be reduced, the limitation on the number of slice file servers can be avoided, and in the case that the number of slice file servers is increased, the performance of the private client in concurrent operation is better, and the performance of the file attribute information obtaining operation can be ensured in the case that the performance of the data concurrent read-write operation is ensured.

[0196] In summary, in the distributed file system provided in the embodiment of the present application, the private client, the master file server and the slice file server cooperatively process file semantic operations such as data read-write, file truncation and file attribute information obtaining by means of the respective corresponding data layout tables, the data layout tables can be used to efficiently manage the file slices, the slices can be allocated to more slice file servers, the performance of the file truncation operation and the file attribute information obtaining operation can be ensured in the case that the performance of the private client in concurrent data read-write operation is improved, and the data consistency can be ensured in the case that the file semantic operations are concurrently executed by means of the version number and the file data amount owner identifier included in the data layout table.

[0197] In a possible implementation, in the case that the private client performs the file semantic operation of deleting a file, the master file server deletes the stored data layout table and file metadata such as file data amount, and the slice file server deletes the stored slice file metadata and data layout information.

[0198] In a possible implementation, the above Figures 4-8The method shown can also be applied to distributed file systems like Lustre. Specifically, in a Lustre file system, files are split across multiple storage nodes. Storage nodes manage the size of the file shards, and shard file servers manage information about the multiple storage nodes containing the shards. Therefore, the master file server modifies the shard file index nodes recorded in the data layout table to the storage nodes. The shard information indicates which storage node manages each shard, while other data in the data layout table remains unchanged. Afterward, the private client, master file server, shard file servers, and object storage devices collaboratively handle file semantic operations based on the data layout table. Figures 4-8 The method shown.

[0199] In another possible implementation, the above Figures 4-8 The method shown can also be applied to systems without private clients, such as network file system (NFS) protocol access file systems. Specifically, in this system, the NFS server accessed by the user equipment is used to cache file metadata included in the main file server, fragment file metadata and data layout information included in the fragment file server, data layout table, etc., in order to ensure data consistency during the execution of file semantic operations and improve the performance of file semantic operations.

[0200] like Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. This computing device can be applied to... Figure 1 In the distributed file system shown, the private client or file server includes a computing device 900 comprising a processor 910, a memory 920, a communication interface 930, and a bus 940. The processor, memory, and communication interface can be interconnected via an internal bus or communicate through other means such as wireless transmission.

[0201] The processor 910 can be constituted by at least one general-purpose processor, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 910 is configured to execute various types of digital storage instructions.

[0202] The memory 920 can be a volatile memory, such as a random access memory (RAM), a dynamic random access memory (DRAM), a static RAM (SRAM), a synchronous dynamic random access memory (SDRAM), a double data rate RAM (DDR), a cache, or the like, or a combination thereof. The memory 920 includes program codes, and the processor 910 can execute the program codes to perform all the steps performed by the private client or the file server. Figures 4-8 The memory 920 includes program codes, and the processor 910 can execute the program codes to perform all the steps performed by the private client or the file server.

[0203] The communication interface 930 can receive an IO request sent by a user device, and return a processing result for the IO request to the user device, which is not limited herein.

[0204] The bus 940 can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, Figure 9 Only one line is used in the middle, but it does not mean that there is only one bus or one type of bus.

[0205] It should be noted that, Figure 9Just one possible implementation of the embodiments of the present application, in actual applications, the computing device 900 can also include more or less components, which are not limited here.

[0206] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0207] As Figure 10 shown, Figure 10 is a structural schematic diagram of a computing device cluster provided by the embodiments of the present application. The computing device cluster includes at least one computing device 900. The memory 920 in one or more computing devices 900 in the computing device cluster can store the same instructions for executing a file processing method provided by the embodiments of the present application.

[0208] In some possible implementations, the memory 920 of one or more computing devices 900 in the computing device cluster can also respectively store partial instructions for executing the file processing method described above. In other words, the combination of one or more computing devices 900 collectively executes the instructions of the file processing method.

[0209] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. Wherein, the network can be a wide area network or a local area network, etc. Figure 11 shows a possible implementation, Figure 11 is a structural schematic diagram of one or more computing devices connected through a network provided by the embodiments of the present application. As Figure 11As shown, three computing devices 900A, 900B and 900C are connected through a network, the computing device 900A includes a processor 910A, a memory 920A, a communication interface 930A and a bus 940A, the computing device 900B includes a processor 910B, a memory 920B, a communication interface 930B and a bus 940B, and the computing device 900C includes a processor 910C, a memory 920C, a communication interface 930C and a bus 940C. Specifically, the communication interface in each computing device is connected with the network. In this kind of possible implementation, the memory 920A in the computing device 900A stores instructions corresponding to the private client. Meanwhile, the memory 920B in the computing device 900B stores instructions corresponding to the master file server, and the memory 920C in the computing device 900C stores instructions corresponding to the shard file server. The computing device 900A, the computing device 900B and the computing device 900C implement the file processing method provided in the embodiments of the present application through communication.

[0210] The embodiments of the present application also provide a computer readable storage medium, which stores computer program instructions. When the program instructions are run by a computing device cluster, the method flow shown in Figures 4-8 is implemented.

[0211] The embodiments of the present application also provide a computer program product, which includes instructions. When run on a computing device cluster, the method flow shown in Figures 4-8 is implemented.

[0212] When the computer program instructions are loaded or executed on a computer, the whole or part of the flow or function according to the embodiments of the present application is produced. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.

[0213] The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer or a data storage node such as a server, data center, etc. containing at least one available medium set. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, high-density digital video disc (digital video disc, DVD), or a semiconductor medium).

[0214] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A file processing method, characterized in that, An application is made to a distributed file system, wherein the distributed file system includes at least one private client and multiple file servers, the multiple file servers include a master file server and multiple shard file servers, the at least one private client includes a first private client, the master file server splits the target file of the first private client into N shard files, the N shard files are managed by M file servers, where N is a positive integer greater than 1, and M is greater than or equal to N, the method includes: The first private client determines the first fragment identifier of the first data to be written, and the first fragment identifier is obtained based on the data offset address of the first data and the length of the first data; The first private client sends the first fragment creation information to the main file server based on the cached first data layout table and the first fragment identifier, wherein the first fragment identifier is greater than the first maximum fragment identifier in the first data layout table; The master file server creates the first fragment based on the first fragment creation information and the stored second data layout table, wherein the first fragment identifier is greater than the second maximum fragment identifier in the second data layout table; The master file server updates the second maximum fragment identifier included in the second data layout table to the first fragment identifier; The master file server sends the updated second data layout table to the first private client, so that the first private client sends the first data to the first shard file server to which the first shard belongs, according to the updated second data layout table.

2. The method according to claim 1, characterized in that, Before the first private client transmits the first shard creation information to the main file server based on the cached first data layout table and the first shard identifier, the method further includes: The first private client determines that the sum of the first data to be written and the second data already written is greater than the split threshold, and sends file split information to the main file server, wherein the split threshold is the maximum amount of data managed by the main file server; The master file server creates the first data layout table based on the file splitting information and sends the split file creation information to the one or more split file servers. The first data layout table includes the first maximum split identifier and the first file data volume owner identifier. The first file data volume owner identifier is used to indicate the second split file server to which the split corresponding to the first maximum split identifier belongs. After receiving the fragmented file creation responses from the multiple fragmented file servers, the main file server sends the first data layout table to the first private client.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The master file server updates the first file data volume owner identifier included in the second data layout table to a second file data volume owner identifier, wherein the second file data volume owner identifier is used to indicate the first shard file server to which the first shard belongs.

4. The method according to claim 3, characterized in that, The method further includes: The main file server sends an identifier clearing message to the second fragment file server; The second shard file server clears the file data owner identifier according to the identifier clearing information, and sends the first cache invalidation information to the multiple private clients through the distributed lock server, so that the first shard file metadata cached on the multiple private clients includes the data volume of multiple shards included in the second shard file server.

5. The method according to claim 4, characterized in that, The method further includes: The master file server sends the identification setting information to the first fragment file server; The first fragment file server sets the owner identifier of the second file data volume according to the identifier setting information, and updates the first fragment file data volume included in the first fragment file server to the second fragment file data volume, wherein the second fragment file data volume is determined according to the first fragment identifier and the data volume of each of the multiple fragments included in the file; The first shard file server sends the second cache invalidation information to the plurality of private clients through the distributed lock server, so as to invalidate the second shard file metadata corresponding to the first shard file server cached on the plurality of private clients. The second shard file metadata includes the data volume of the plurality of shards included in the first shard file server.

6. The method according to claim 1, characterized in that, The at least one private client further includes a second private client, and the method further includes: The second private client determines the second fragment identifier of the third data to be written, which is obtained based on the data offset address of the third data and the length of the third data; The second private client sends the second fragment creation message to the main file server according to the cached third data layout table and the second fragment identifier. The third data layout table includes a third maximum fragment identifier and a first bit. The third maximum fragment identifier is greater than or equal to the second fragment identifier. The bit value of the first bit indicates that the data included in the fragment corresponding to the second fragment identifier is invalid. The master file server sends data clearing information to the third file server to which the second fragment belongs, based on the second fragment creation information and the stored fourth data layout table. The fourth data layout table includes a fourth maximum fragment identifier and a second bit. The fourth maximum fragment identifier is greater than or equal to the second fragment identifier, and the bit value of the second bit indicates that the data in the fragment corresponding to the second fragment identifier is invalid. In response to the data clearing information, the third fragment file server clears the data included in the second fragment and sends a data clearing response to the main file server.

7. The method according to claim 1, characterized in that, The at least one private client also includes a third private client, and the method further includes: The third private client determines the third fragment identifier of the fourth data to be written, and the third fragment identifier is obtained based on the data offset address of the fourth data and the length of the fourth data; The third private client sends the version number of the fifth data layout table and the third fragment identifier to the fourth fragment file server. The fifth data layout table includes a fifth maximum fragment identifier and a third bit. The fifth maximum fragment identifier is greater than or equal to the third fragment identifier. The bit value of the third bit indicates that the data included in the fragment corresponding to the third fragment identifier is valid. The version number of the fifth data layout table is the version number of the fifth data layout table before the file truncation operation is performed. The main file server sends the version number of the sixth data layout table to the fourth fragment file server. The version number of the sixth data layout table is the version number after the file is truncated. The version number of the sixth data layout table is greater than the version number of the fifth data layout table. The fourth shard file server determines that the version number of the fifth data layout table is different from the version number of the sixth data layout table, and returns a write data error to the third private client.

8. The method according to claim 7, characterized in that, The at least one private client further includes a fourth private client, and the method further includes the following steps before the third private client sends the version number of the fifth data layout table and the third fragment identifier to the fourth fragment file server: The fourth private client sends the file truncation operation to the main file server; The main file server sends version number clearing information to one or more fragment file servers corresponding to the file truncation operation according to the file truncation operation; The main file server updates the version number of the cached seventh data layout table to the version number of the sixth data layout table according to the file truncation operation. The version number of the seventh data layout table is the version number of the file before the file truncation operation was performed.

9. The method according to claim 8, characterized in that, The method further includes: According to the file truncation operation, the master file server updates the third file data volume owner identifier included in the seventh data layout table to the fourth file data volume owner identifier included in the sixth data layout table. The third file data volume owner identifier is used to indicate the fifth shard file server to which the shard corresponding to the seventh maximum shard identifier belongs, and the fourth file data volume owner identifier is used to indicate the sixth shard file server to which the shard corresponding to the sixth maximum shard identifier belongs. According to the file truncation operation, the master file server modifies the first bit value of one or more fourth bits in the seventh data layout table to the second bit value. The fourth bit is used to indicate the fragment corresponding to the first file truncation operation. The first bit value indicates that the data included in the fragment is valid, and the second bit value indicates that the data included in the fragment is invalid.

10. The method according to claim 1, characterized in that, The method further includes: The first private client obtains the operation and cached eighth data layout table based on the first attribute, as well as the cached data volume of each of the multiple file servers, and determines the first file attribute information. The cached data volume of the first file server included in the first file server is valid. The first file attribute information includes at least one of the following: the data volume of the target file, the creation time of the target file, and the modification time of the target file.

11. The method according to claim 1, characterized in that, The method further includes: The first private client, based on the second attribute acquisition operation, the cached ninth data layout table, and the cached data volume of each of the multiple file servers' shard files, sends data layout table acquisition information to the main file server and data volume acquisition information to the first shard file server. The cached data volume of the first shard file server's shard files is invalid. The first private client, based on the second data layout table sent by the main file server and the target shard file data volume sent by the first shard file server, determines second file attribute information, which includes at least one of the following: the target file data volume, the target file's creation time, and the target file's modification time.

12. A file system, characterized in that, The system includes at least one private client and multiple file servers. The multiple file servers include a main file server and multiple shard file servers. The at least one private client includes a first private client. The main file server splits the target file of the first private client into N shard files. The N shard files are managed by M file servers, where N is a positive integer greater than 1, and M is greater than or equal to N. The first private client is used to determine the first fragment identifier of the first data to be written, the first fragment identifier being obtained based on the data offset address of the first data and the length of the first data; The first private client is further configured to send the first fragment creation information to the main file server based on the cached first data layout table and the first fragment identifier; The main file server is used to create the first fragment based on the first fragment creation information and the stored second data layout table, wherein the first fragment identifier is greater than the second maximum fragment identifier in the second data layout table; The master file server is also used to update the second maximum fragment identifier included in the second data layout table to the first fragment identifier; The master file server is further configured to send the updated second data layout table to the first private client, so that the first private client sends the first data to the first shard file server to which the first shard belongs, according to the updated second data layout table.

13. The system according to claim 12, characterized in that, The first private client is further configured to determine that the sum of the data volume of the first data to be written and the second data already written is greater than the split threshold, and send file split information to the main file server, wherein the split threshold is the maximum data volume managed by the main file server; The master file server is further configured to create the first data layout table based on the file splitting information, and send the split file creation information to the one or more split file servers. The first data layout table includes the first maximum split identifier and the first file data volume owner identifier. The first file data volume owner identifier is used to indicate the second split file server to which the split corresponding to the first maximum split identifier belongs. The master file server is further configured to send the first data layout table to the first private client after receiving the fragmented file creation response sent by each of the plurality of fragmented file servers.

14. The system according to claim 12 or 13, characterized in that, The master file server is further configured to update the first file data volume owner identifier included in the second data layout table to a second file data volume owner identifier, wherein the second file data volume owner identifier is used to indicate the first shard file server to which the first shard belongs.

15. The system according to claim 14, characterized in that, The main file server is also used to send identifier clearing information to the second fragment file server; The second shard file server is used to clear the file data owner identifier according to the identifier clearing information, and send the first cache invalidation information to the multiple private clients through the distributed lock server, so as to invalidate the first shard file metadata included in the second shard file server cached on the multiple private clients. The first shard file metadata includes the data volume of multiple shards included in the second shard file server.

16. The system according to claim 15, characterized in that, The main file server is also used to send the identification setting information to the first fragment file server; The first fragment file server is further configured to set the owner identifier of the second file data volume according to the identifier setting information, and update the first fragment file data volume included in the first fragment file server to the second fragment file data volume, wherein the second fragment file data volume is determined according to the first fragment identifier and the data volume of each of the multiple fragments included in the file; The first shard file server is further configured to send the second cache invalidation information to the plurality of private clients through the distributed lock server, so as to invalidate the second shard file metadata corresponding to the first shard file server cached on the plurality of private clients, wherein the second shard file metadata includes the data volume of the plurality of shards included in the first shard file server.

17. The system according to claim 12, characterized in that, The at least one private client also includes a second private client; The second private client is used to determine the second fragment identifier of the third data to be written, the second fragment identifier being obtained based on the data offset address of the third data and the length of the third data; The second private client is further configured to send the second shard creation message to the main file server based on the cached third data layout table and the second shard identifier. The third data layout table includes a third maximum shard identifier and a first bit. The third maximum shard identifier is greater than or equal to the second shard identifier. The bit value of the first bit indicates that the data included in the shard corresponding to the second shard identifier is invalid. The master file server is further configured to send data clearing information to the third file server to which the second fragment belongs, based on the second fragment creation information and the stored fourth data layout table. The fourth data layout table includes a fourth maximum fragment identifier and a second bit. The fourth maximum fragment identifier is greater than or equal to the second fragment identifier, and the bit value of the second bit indicates that the data included in the fragment corresponding to the second fragment identifier is invalid. The third shard file server is configured to, in response to the data clearing information, clear the data included in the second shard and send a data clearing response to the main file server.

18. The system according to claim 12, characterized in that, The at least one private client also includes a third private client; The third private client is used to determine the third fragment identifier of the fourth data to be written, the third fragment identifier being obtained based on the data offset address of the fourth data and the length of the fourth data; The third private client is also used to send the version number of the fifth data layout table and the third fragment identifier to the fourth fragment file server. The fifth data layout table includes a fifth maximum fragment identifier and a third bit. The fifth maximum fragment identifier is greater than or equal to the third fragment identifier. The bit value of the third bit indicates that the data included in the fragment corresponding to the third fragment identifier is valid. The version number of the fifth data layout table is the version number of the fifth data layout table before the file truncation operation is performed. The main file server is also used to send the version number of the sixth data layout table to the fourth fragment file server, wherein the version number of the sixth data layout table is the version number after the file is truncated, and the version number of the sixth data layout table is greater than the version number of the fifth data layout table. The fourth shard file server is used to determine that the version number of the fifth data layout table and the version number of the sixth data layout table are different, and then returns a write data error to the third private client.

19. The system according to claim 18, characterized in that, The at least one private client also includes a fourth private client; The fourth private client is used to send the file truncation operation to the main file server; The main file server is also configured to send version number clearing information to one or more fragment file servers corresponding to the file truncation operation according to the file truncation operation; The main file server is further configured to update the version number of the cached seventh data layout table to the version number of the sixth data layout table according to the file truncation operation, wherein the version number of the seventh data layout table is the version number of the file before the file truncation operation is performed.

20. The system according to claim 19, characterized in that, The master file server is further configured to update the third file data volume owner identifier included in the seventh data layout table to the fourth file data volume owner identifier included in the sixth data layout table according to the file truncation operation, wherein the third file data volume owner identifier is used to indicate the fifth shard file server to which the shard corresponding to the seventh maximum shard identifier belongs, and the fourth file data volume owner identifier is used to indicate the sixth shard file server to which the shard corresponding to the sixth maximum shard identifier belongs. The master file server is further configured to modify the first bit value of one or more fourth bits included in the seventh data layout table to a second bit value according to the file truncation operation, wherein the fourth bit is used to indicate the fragment corresponding to the first file truncation operation, the first bit value indicates that the data included in the fragment is valid, and the second bit value indicates that the data included in the fragment is invalid.

21. The system according to claim 12, characterized in that, The first private client is further configured to obtain the operation and cached eighth data layout table and the cached data volume of each of the multiple file servers based on the first attribute, and determine the first file attribute information, wherein the cached data volume of the first file server includes the first file segment data volume, and the first file attribute information includes at least one of the following: the data volume of the target file, the creation time of the target file, and the modification time of the target file.

22. The system according to claim 12, characterized in that, The first private client is further configured to obtain data layout table acquisition information from the second attribute, the cached ninth data layout table, and the cached data volume of each of the plurality of file servers, and send data volume acquisition information to the main file server, wherein the cached data volume of the first fragmented file server is invalid; the first private client determines second file attribute information based on the second data layout table sent by the main file server and the target fragmented file data volume sent by the first fragmented file server, wherein the second file attribute information includes at least one of the following: the target file data volume, the creation time of the target file, and the modification time of the target file.

23. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

24. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to implement the method as described in any one of claims 1-11.

25. A computer-readable storage medium, characterized in that, Includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

Images