File cloning method and device of distributed file system and storage medium

Abstract

The embodiment of the invention provides a file cloning method and device of a distributed file system and a storage medium. In the file cloning method, a client obtains fragment identifiers of a plurality of source file fragments corresponding to a source file from a source management node, and generates respective fragment identifiers of a plurality of target file fragments according to the fragment identifiers of the plurality of source file fragments. Based on the fragment identifier of the source file fragment and the fragment identifier of the destination file fragment, the client may send a hard link establishment request to the storage cluster. The storage cluster can establish a corresponding relationship between the fragment identifier of the target file fragment and the data block according to the corresponding relationship between the source file fragment and the data block. And the destination management node can create a destination file under the request of the client, establish an association relationship between the destination file and the respective fragment identifiers of the plurality of destination file fragments, and publish the destination file. Furthermore, the cloning of the source file is realized without copying and migrating the data block, and the complexity of file cloning is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technical field, and particularly relates to a file cloning method, device and storage medium of a distributed file system. BACKGROUND

[0002] The distributed file system is composed of a client, a master server and a chunk server. The master server plays a vital role in the distributed file system, and is mainly responsible for storing and managing metadata files of the file system. The metadata files are key information used by users and the distributed file system to locate and access actual file data.

[0003] At present, the distributed system can expand the master server based on a federation mechanism to form a structure in which multiple master servers work cooperatively. The federation mechanism allows different master servers to share data or resources, so that data can be transparently accessed among multiple independent master servers. Different master servers can share metadata files by cloning the metadata files. In some existing methods, in order to ensure the reliability and data consistency of the cloning operation, a distributed transaction is usually initiated among multiple master servers to realize the cloning of the metadata files. However, the file cloning method with the transaction mechanism has high complexity. Therefore, a new solution is needed. SUMMARY

[0004] Aspects of the present application provide a file cloning method, device and storage medium of a distributed file system, which realizes file cloning by hard linking data blocks and a newly created target file, and reduces the complexity of file cloning.

[0005] The embodiment of the application provides a file cloning method of a distributed file system, applicable to a client, comprising: in response to a cloning request for a source file in a source management node, obtaining respective shard identifiers of a plurality of source file shards corresponding to the source file from the source management node; the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored in a plurality of storage nodes in a storage cluster; sending a file creation request to a target management node, the file creation request being used to make the target management node create a target file; generating respective shard identifiers of a plurality of target file shards according to the shard identifiers of the plurality of source file shards; sending a hard link establishment request to the storage cluster according to the respective shard identifiers of the plurality of source file shards and the respective shard identifiers of the plurality of target file shards, the hard link establishment request being used to make the storage cluster establish a correspondence between the respective shard identifiers of the plurality of target file shards and the plurality of data blocks according to a correspondence between the plurality of source file shards and the plurality of data blocks; when the storage cluster returns a hard link establishment success message, sending a cloning submission instruction to the target management node according to the respective shard identifiers of the plurality of target file shards, the cloning submission instruction being used to make the target management node establish an association between the target file and the respective shard identifiers of the plurality of target file shards and publish the target file; and after receiving a submission response message returned by the target management node, taking the target file as a cloned file of the source file.

[0006] Optionally, after taking the target file as the cloned file of the source file, the method further comprises: in response to an append write request for the target file, sending a write file request to the target management node according to to-be-written data corresponding to the append write request, the write file request being used to make the target management node allocate a first data block in a target storage node for the to-be-written data and generate a shard identifier of a first target file shard corresponding to the first data block in the target storage node; receiving identifier information of the target storage node and position information of the first data block in the target storage node returned by the target management node; and writing the to-be-written data corresponding to the append write request into the first data block in the target storage node according to the identifier information of the target storage node and the position information of the first data block in the target storage node, the first data block being different from a plurality of data blocks corresponding to the plurality of source file shards.

[0007] Optionally, after the destination file is cloned from the source file, the method further comprises: in response to a deletion request for any second destination file segment of the destination file, determining a second data block corresponding to the second destination file segment; if the second data block has no corresponding relationship with other file segments except the second destination file segment, deleting the second destination file segment and the second data block; if the second data block has a corresponding relationship with other file segments except the second destination file segment, deleting the second destination file segment and retaining the second data block.

[0008] Optionally, the method further comprises: in response to a modification request for the destination file, determining a segment identifier of any third destination file segment to be modified; and sending the segment identifier of the third destination file segment and the modification request to a storage node where the third destination file segment is located, so that the storage node modifies a third data block according to the corresponding relationship between the segment identifier of the third destination file segment and the third data block.

[0009] Optionally, in response to a cloning request for a source file in a source management node, the method of obtaining the segment identifiers of the source file segments corresponding to the source file from the source management node comprises: in response to the cloning request, configuring an interface parameter of a cloning preparation interface of a cloning tool according to a source file identifier and a preset destination file identifier; calling the cloning preparation interface of the cloning tool according to the configured interface parameter, so that the cloning tool reads a file corresponding to the source file identifier from the source management node as the source file to be cloned and obtains the segment identifiers of the source file segments corresponding to the source file; and sending a file creation request to a destination management node, including: sending a file creation request to the destination management node according to the destination file identifier by using the cloning tool, so that the destination management node creates the destination file according to the destination file identifier.

[0010] Optionally, after the destination file is cloned from the source file, the method further comprises: in response to an access request for the destination file, sending a query request to the destination management node, so that the destination management node returns the segment identifiers of at least one destination file segment corresponding to the destination file; and sending an access request to a corresponding storage node according to the segment identifiers of the at least one destination file segment, so that the corresponding storage node accesses a corresponding data block according to the corresponding relationship between the segment identifiers of the at least one destination file segment and the data block.

[0011] The application further provides a file cloning method of a distributed file system, applicable to a destination management node, comprising: receiving a file creation request sent by a client; creating a destination file and returning a creation success message to the client, the creation success message being used to make the client generate a shard identifier of each of a plurality of destination file shards according to a shard identifier of each of a plurality of source file shards and send a hard link establishment request to a storage cluster according to the shard identifier of each of the plurality of source file shards and the shard identifier of each of the plurality of destination file shards; the plurality of source file shards belong to a source file on a source management node, the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in the storage cluster; the hard link request is used to trigger the storage cluster to establish a correspondence between the shard identifier of each of the plurality of destination file shards and the plurality of data blocks according to a correspondence between the plurality of source file shards and the plurality of data blocks; in response to a cloning submission instruction sent by the client according to the shard identifier of each of the plurality of destination file shards, establishing an association between the destination file and the shard identifier of each of the plurality of destination file shards and publishing the destination file; and returning a submission response message to the client, the submission response message being used to make the client take the destination file as a cloned file of the source file.

[0012] The application further provides a file cloning method of a distributed file system, applicable to any storage node in a storage cluster, comprising: receiving a hard link establishment request sent by a client, the hard link request being sent by the client after a destination file is created on a destination management node; according to the hard link establishment request, obtaining a shard identifier of each of at least one source file shard and a shard identifier of each of at least one destination file shard, the shard identifier of each of the at least one destination file shard being generated according to a shard identifier of the at least one source file shard; the at least one source file shard belongs to a source file on a source management node, and the at least one source file shard corresponds to at least one data block on a storage node; according to a correspondence between the at least one source file shard and the at least one data block, establishing a correspondence between the shard identifier of each of the at least one destination file shard and a data block corresponding to each of the at least one source file shard; and sending a hard link establishment success message to the client, the hard link establishment success message being used to make the client send a cloning submission instruction to the destination management node according to a shard identifier of each of a plurality of destination file shards after receiving hard link establishment success messages returned by a plurality of storage nodes in the storage cluster, the cloning submission instruction being used to make the destination management node establish an association between the destination file and the shard identifier of each of the plurality of destination file shards and publish the destination file.

[0013] Optionally, the establishing of the correspondence between the respective file slice identifier of each of the at least one target file slice and the data block corresponding to each of the at least one source file slice comprises: for any target file slice in the at least one target file slice, determining a source file slice corresponding to the target file slice according to the correspondence between the at least one source file slice and the at least one target file slice; determining a target data block corresponding to the source file slice as a target data block corresponding to the target file slice according to the correspondence between the at least one source file slice and the at least one data block; and creating a first index node according to metadata information of the target data block, the metadata information at least including address information of the target data block on a storage node, and establishing a correspondence between the first index node and the file slice identifier of the target file slice.

[0014] Optionally, the establishing of the correspondence between the respective file slice identifier of each of the at least one target file slice and the data block corresponding to each of the at least one source file slice comprises: for any target file slice in the at least one target file slice, determining a source file slice corresponding to the target file slice according to the correspondence between the at least one source file slice and the at least one target file slice; if the source file slice is not a cloned file slice, determining a second index node corresponding to the source file slice, the second index node being used to record address information of a target data block corresponding to the source file slice on a storage node; and establishing a correspondence between the second index node and the file slice identifier of the target file slice.

[0015] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; and the processor is used to execute the one or more computer instructions to execute the steps in the method provided by the embodiment of the present application.

[0016] The embodiment of the present application further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the steps in the method provided by the embodiment of the present application.

[0017] The embodiment of the present application further provides a computer program product, comprising: a computer program / instruction, which is executed by a processor to implement the steps in the method provided by the embodiment of the present application.

[0018] In the file cloning method provided by the embodiments of the present application, when cloning a source file on a source management node, the client can obtain the shard identifiers of the plurality of source file shards corresponding to the source file from the source management node, and generate the shard identifiers of the plurality of destination file shards respectively according to the shard identifiers of the plurality of source file shards. Based on the shard identifiers of the source file shards and the shard identifiers of the destination file shards, the client can send a hard link establishment request to the storage cluster. The storage cluster can establish the correspondence between the shard identifiers of the destination file shards and the data blocks according to the correspondence between the source file shards and the data blocks. The destination management node can create the destination file at the request of the client, establish the association between the destination file and the shard identifiers of the plurality of destination file shards respectively, and publish the destination file. Further, the access operation on the destination file on the destination management node can be linked to the data blocks corresponding to the source file, so that the cloning of the source file is realized without copying and migrating the data blocks, and the complexity of the file cloning is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0020] Figure 1 The structural schematic diagram of the distributed file system provided by an illustrative embodiment of the present application is shown in FIG. 1.

[0021] Figure 2 The signaling interaction timing diagram of each end in the distributed file system provided by an illustrative embodiment of the present application is shown in FIG. 2.

[0022] Figure 3 The flowchart of the file cloning method provided by an illustrative embodiment of the present application when executed on the client side is shown in FIG. 3.

[0023] Figure 4 The flowchart of the file cloning method provided by an illustrative embodiment of the present application when executed on the destination management node side is shown in FIG. 4.

[0024] Figure 5 The flowchart of the file cloning method provided by an illustrative embodiment of the present application when executed on any storage node side is shown in FIG. 5.

[0025] Figure 6 The structural schematic diagram of the electronic device provided by an illustrative embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0027] The terms used in the embodiments of the present application are merely for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0028] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0029] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that the product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such product or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the product or system including the element.

[0030] The distributed system can expand the management node based on the federation mechanism to form a structure in which multiple management nodes work cooperatively. The federation mechanism allows different management nodes to share data or resources, so that data can be transparently accessed between multiple independent management nodes. Different management nodes can share metadata files by cloning the metadata files.

[0031] In a distributed file system, a federation mechanism of a master server refers to an extended mode of the master server, which allows different independent master servers to work cooperatively, thereby enhancing the scalability, fault tolerance and geographical distribution capability of the distributed file system. The federation allows different master servers to share data or resources, so that the data can be transparently accessed among multiple independent systems. Further, the load can be dispersed to multiple master servers through the federation mechanism, thereby reducing the pressure on a single master server and improving the processing capability and scalability of the entire distributed file system. When a master server fails, other master servers can continue to serve, thereby reducing the impact of single-point failure.

[0032] To implement the metadata sharing operation among multiple master servers, a clone operation of a metadata file can be performed among the multiple master servers. The clone operation does not move the underlying data corresponding to the metadata file, and the source file to be cloned and the destination file after cloning can share the data blocks of the source file at the moment of cloning, thereby reducing the space overhead.

[0033] In some methods, to ensure the reliability and data consistency of the clone operation, a transaction mechanism can be used to perform the clone operation of the metadata file among the multiple master servers. When it is necessary to clone a file among different master servers, the distributed file system can use a two-phase commit method or other distributed transaction protocol to ensure that all the master servers participating in the clone operation either all complete the operation or none of them completes the operation, so as to maintain the data consistency. In the first phase of the transaction, a coordinator can initiate a transaction and send a request for preparing the clone to the destination master servers. Each master server receiving the request can perform a preparation operation before the clone and return a preparation confirmation message to the coordinator. After the coordinator receives the preparation confirmation messages returned by all the destination master servers, the coordinator enters the second phase of the transaction. In the second phase, the coordinator can send a commit instruction to the destination master servers, and the destination master servers receiving the commit instruction can obtain the link information of the metadata file from the source master server, save the link information locally, and return a commit confirmation message to the coordinator. After the coordinator receives the commit confirmation messages returned by all the destination master servers, the coordinator determines that the transaction is completed. The file cloning method based on the transaction mechanism has high complexity. In addition, since the link information of the metadata file is stored on the destination master servers, any modification operation on the metadata by the destination master servers needs to be performed on the source master server. When multiple destination master servers modify the metadata, the access amount to the source master server will increase, and the throughput performance of the source master server will become the bottleneck of the metadata modification service. Accordingly, when the master servers are horizontally expanded using the federation mechanism, the access pressure borne by the source master server will limit the freedom of horizontal expansion of the master servers.

[0034] In some embodiments of the present application, a solution is provided to solve the above technical problems. The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0035] Figure 1 is a structural schematic diagram of a distributed file system provided by an exemplary embodiment of the present application. As shown in the figure, the distributed file system 100 can include a client 10, a plurality of management nodes 20, and a storage cluster 30. Figure 1

[0036] Among them, the management node is responsible for maintaining the metadata file of the file in the distributed file system 100. The metadata file is mainly used to describe at least one of the file name, the file size, the file attribute (such as the creation time, the modification time), the storage location of the file (for example, the storage node where the file is located), and the directory structure. When the client 10 requests to access a certain file, the management node can route the access request of the client 10 to the corresponding storage node according to the maintained metadata information.

[0037] In the distributed file system 100, the plurality of management nodes 20 form a federation and can work cooperatively to enhance the scalability, fault tolerance and geographical distribution capability of the distributed file system. When cloning files among the plurality of management nodes, the starting management node of the data flow is described as the source management node, such as the source management node 201 shown in Figure 1 , and the terminal management node of the data flow is described as the destination management node, such as the destination management node 202 shown in Figure 1 .

[0038] The storage cluster 30 can include a plurality of storage nodes. Any storage node is an entity in the distributed file system 100 that is actually used to store data, responsible for receiving data blocks allocated from the management node and persistently storing them on the local hard disk or other storage medium. In the distributed file system 100, any file can be divided into a plurality of file fragments, and the data blocks corresponding to the plurality of file fragments can be stored in a plurality of storage nodes in the storage cluster 30, so as to improve the data access speed and throughput through parallel access mechanism.

[0039] ​In the embodiment, a file fragment refers to a logical fragment obtained by logically dividing a file. A data block refers to a physical fragment, which is the smallest physical unit for data storage and access on a storage node. In a storage medium such as a hard disk, a solid state disk, or a memory, data is read into and read out of the storage medium in the form of a block. One file fragment can correspond to multiple data blocks. In the file system of any storage node, each file fragment has an index node, which is used to describe the mapping relationship between the file fragment and the data blocks corresponding to the file fragment. For example, the index node can include at least the size of the file fragment, the owner of the file fragment, the permissions (read, write, and execute) of the file, the last access and modification time of the file fragment, the number of hard links of the file fragment, and the location information (for example, the pointer of the data block) of the data blocks corresponding to the file fragment.

[0040] When the client 10 accesses a file, the client 10 can query the fragment identifiers of the file fragments corresponding to the file through any management node to obtain a set of fragment identifiers of the file fragments. For each file fragment, the client 10 can access the corresponding storage node according to the fragment identifier of the file fragment. The storage node can determine the index node corresponding to the file fragment according to the fragment identifier of the file fragment carried in the access request of the client 10, and read the data blocks corresponding to the file fragment based on the index node.

[0041] In the embodiment, to expand the service capability of the source management node 201, the metadata file on the source management node 201 can be cloned into the destination management node 202. The metadata file cloned on the source management node 201 is described as a source file. To facilitate distinction, the file fragments of the source file are described as source file fragments, and the file fragments of the cloned destination file are described as destination file fragments.

[0042] In the distributed file system, the client 10 is mainly used to: in response to a cloning request for a source file, acquire the respective fragment identifiers of the multiple source file fragments corresponding to the source file to be cloned from the source management node 201. The cloning request for the source file can be triggered by a user or triggered by a certain system event in the distributed file system 100, which is not limited in the embodiment. For example, the system event can be an event in which the load pressure of the source management node 201 is greater than a certain set threshold. The multiple source file fragments of the source file correspond to multiple data blocks, one source file fragment can correspond to one data block, and the multiple data blocks are stored on multiple storage nodes in the storage cluster 30. After the client 10 acquires the respective fragment identifiers of the multiple source file fragments, the client 10 can send a file creation request to the destination management node 202.

[0043] The destination management node 202 receives the file creation request sent by the client 10, and can create a destination file and return a creation success message to the client 10. After receiving the creation success message of the destination file, the client 10 can generate the shard identifiers of the plurality of destination file shards according to the shard identifiers of the plurality of source file shards.

[0044] Before the cloning operation is completed, the destination file created by the destination management node 202 is a temporary file, which is stored in a directory invisible to the client 10, so as to be isolated from the formal file and avoid confusion. Before the cloning operation is submitted, the file name of the destination file can be expressed as: dstfilename.tmp, and tmp is used to identify the temporary attribute of the destination file.

[0045] In this embodiment, for ease of description, the shard identifier of the source file shard is marked as ChunkID, the shard identifier of the source file shard after adding a prefix representing "source" is marked as src-ChunkID, and the shard identifier of the destination file shard is marked as dst-ChunkID. The dst-ChunkID is generated according to the src-ChunkID. Optionally, when the dst-ChunkID is generated according to the src-ChunkID, the prefix src of the src-ChunkID can be modified to the prefix dst used to represent the destination. In this way, on the one hand, the dst-ChunkID can be quickly generated, and on the other hand, the distributed file system 100 can distinguish different file shards through the difference of the prefix part, and can express the correspondence between the src-ChunkID and the dst-ChunkID through the ChunkID of the non-prefix part.

[0046] After the client 10 obtains the shard identifiers of the plurality of destination file shards, the client 10 can send a hard link establishment request to the storage cluster 30 according to the shard identifiers of the plurality of source file shards and the shard identifiers of the plurality of destination file shards.

[0047] In this embodiment, the data blocks corresponding to the source file shards are stored in the storage nodes, and the correspondence between the source file shards and the storage nodes can be saved in the metadata managed by the source management node 201. The client 10 can read the correspondence between the source file shards and the storage nodes, and send a hard link establishment request to each storage node corresponding to the source file shard according to the correspondence. For example, the storage nodes corresponding to the source file shard A1 are the storage node S1 and the storage node S2, and the client 10 can send a hard link establishment request to the storage node S1 and the storage node S2 according to the shard identifier of the source file shard A1. For another example, the storage node corresponding to the source file shard A2 is the storage node S3, and the client 10 can send a hard link establishment request to the storage node S3 according to the shard identifier of the source file shard A2.

[0048] Accordingly, for any storage node in the storage cluster 30, the storage node is mainly configured to receive the hard link establishment request sent by the client 10, and obtain the shard identifier of each of the at least one source file shard and the shard identifier of each of the at least one destination file shard according to the hard link establishment request. The shard identifier of each of the at least one destination file shard is generated according to the shard identifier of each of the at least one source file shard, so that the shard identifier of the destination file shard and the shard identifier of the source file shard have a one-to-one correspondence. The hard link establishment request can carry the shard identifier of each of the at least one source file shard and the shard identifier of each of the at least one destination file shard. The storage node can establish the correspondence between the shard identifier of each of the at least one destination file shard and the data block corresponding to each of the at least one source file shard according to the correspondence between the at least one source file shard and the at least one data block.

[0049] The following will continue to take any storage node as an example to exemplarily illustrate the optional implementation of the storage node establishing the correspondence between the shard identifier of each of the at least one destination file shard and the data block corresponding to each of the at least one source file shard according to the correspondence between the at least one source file shard and the at least one data block.

[0050] In some exemplary embodiment A1, for any destination file shard dst-C1 in the at least one destination file shard, the storage node can determine the source file shard src-C1 corresponding to the destination file shard dst-C1 according to the correspondence between the at least one source file shard and the at least one destination file shard, and determine the target data block corresponding to the source file shard src-C1 as the target data block corresponding to the destination file shard dst-C1 according to the correspondence between the at least one source file shard and the at least one data block. After determining the target data block, the storage node can create a first index node according to the metadata information of the target data block. The metadata information at least includes address information of the target data block on the storage node. In some embodiments, in addition to the address information, the metadata information can also include at least one of the file size, the owner, the file permission (read, write, and execute), and the latest access and modification time of the destination file shard dst-C1. After the storage node generates the first index node, the storage node can establish the correspondence between the first index node and the shard identifier of the destination file shard.

[0051] In this embodiment, the first inode created by the storage node for the destination file fragment dst-C1 is different from the inode of the source file fragment src-C1, and the two different inodes can point to the same target data block, so that the access operations on the source file fragment src-C1 and the destination file fragment dst-C1 can be linked to the target data block, and the cloning of the source file fragment is realized without copying the target data block.

[0052] In some other exemplary embodiments A2, for any destination file fragment dst-C1 in the at least one source file fragment, according to the correspondence between the at least one source file fragment and the at least one destination file fragment, the source file fragment src-C1 corresponding to the destination file fragment dst-C1 is determined, and it is judged whether the source file fragment src-C1 is a cloned file fragment. If the source file fragment src-C1 is not a cloned file fragment, the storage node can determine the second inode corresponding to the source file fragment src-C1, and the second inode is used to record the address information of the target data block corresponding to the source file fragment src-C1 on the storage node; the correspondence between the second inode and the fragment identifier of the destination file fragment is established. If the source file fragment src-C1 is a cloned file fragment, the storage node can create a new inode corresponding to the target data block according to the metadata information of the target data block, instead of sharing the inode corresponding to the source file fragment src-C1, so as to avoid sharing the inode of the cloned file fragment and reduce the resource management overhead.

[0053] In the storage cluster 30, after establishing the correspondence between the fragment identifier of each of the at least one destination file fragment and the data block corresponding to each of the at least one source file fragment based on the above embodiments, the storage node can send a hard link establishment success message to the client 10.

[0054] The client 10 can send a cloning submission instruction to the destination management node 202 according to the fragment identifiers of the plurality of destination file fragments when the plurality of storage nodes in the storage cluster 30 return the hard link establishment success message. The plurality of storage nodes in the storage cluster 30 refer to the storage nodes to which the client 10 sends the hard link establishment request.

[0055] The destination management node 202 can respond to the clone commit instruction sent by the client 10, establish the association between the destination file and the respective shard identifiers of the plurality of destination file shards, and publish the destination file. After publishing the destination file, the destination management node 202 can return a commit response message to the client 10. The publishing of the destination file refers to providing the temporarily created destination file to external users for access or use, that is, setting the destination file as a file visible to the client 10. Alternatively, the destination file can be created in a temporary file directory that is not visible. After receiving the clone commit instruction of the client 10, the destination management node 202 can migrate the destination file to the user directory tree specified by the client 10 to publish the destination file to the client 10.

[0056] After receiving the commit response message returned by the destination management node 202, the client 10 publishes the destination file as the cloned file of the source file.

[0057] In this embodiment, when cloning the source file on the source management node, the client can obtain the shard identifiers of the plurality of source file shards corresponding to the source file from the source management node, and generate the respective shard identifiers of the plurality of destination file shards according to the shard identifiers of the plurality of source file shards. Based on the shard identifiers of the source file shards and the shard identifiers of the destination file shards, the client can send a hard link establishment request to the storage cluster. The storage cluster can establish the correspondence between the shard identifiers of the destination file shards and the data blocks according to the correspondence between the source file shards and the data blocks. The destination management node can create the destination file at the request of the client, establish the association between the destination file and the respective shard identifiers of the plurality of destination file shards, and publish the destination file. Further, the access operation on the destination file on the destination management node can be linked to the data blocks corresponding to the source file, thereby realizing the cloning of the source file without copying and migrating the data blocks, and reducing the complexity of file cloning.

[0058] In some optional embodiments, a cloning tool for cloning files is run on the client 10, which is in the form of a software development kit (SDK) and runs on the client 10 or other devices outside the client 10, which is not limited in this embodiment. The client 10 can realize the cloning of files between a plurality of management nodes by calling the interface provided by the cloning tool. Alternatively, the interface provided by the cloning tool can include a clone preparation interface. For example, the interface for cloning metadata can be described as: PrepareCloneFileMeta(srcfile, dstfile). In the interface parameters of the clone preparation interface, srcfile is used to configure the file identifier of the source file to be cloned, and dstfile is used to configure the file identifier of the destination file after cloning.

[0059] Optionally, when the client 10 acquires the respective shard identifiers of the plurality of source file shards corresponding to the source file to be cloned from the source management node 201 in response to the clone request for the source file, the client 10 can respond to the clone request and configure the interface parameters of the clone preparation interface of the clone tool according to the source file identifier of the source file and the destination file identifier preset for the destination file. The file identifier of the source file can be the file name of the source file, and correspondingly, the file name of the destination file can also be the file name preset for the destination file. For example, the configured clone preparation interface can be represented as: clone(srcfilename, dstfilename). Wherein, srcfilename is the file name of the source file, and dstfilename is the file name of the destination file. After the configuration is completed, the client 10 can invoke the clone preparation interface of the clone tool according to the configured interface parameters, so as to read the file corresponding to the source file identifier from the source management node 201 as the source file to be cloned and acquire the respective shard identifiers of the plurality of source file shards corresponding to the source file.

[0060] Correspondingly, when the client 10 sends the file creation request to the destination management node 202, the client 10 can use the clone tool to send the file creation request to the destination management node 202 according to the destination file identifier. Optionally, after the destination management node 202 creates the destination file and returns the creation success message to the client 10, the client 10 can use the clone tool to generate the shard identifiers of the destination file shards. When the clone tool generates the respective shard identifiers of the plurality of destination file shards corresponding to the destination file according to the shard identifiers of the plurality of source file shards, the clone tool can add the prefix dst (destination) indicating the destination in the shard identifiers of the source file shards as the shard identifiers of the destination file shards. In addition, the client 10 can also add the prefix src (source) indicating the source to the shard identifiers of the plurality of source file shards for differentiation.

[0061] In this embodiment, the client 10 can quickly request the destination management node 202 to create the destination file based on the pre-built code library and modules provided by the SDK, and use the clone tool to generate the shard identifiers of the destination file, so that the client 10 does not need to be customized and developed, and the cloning efficiency can be effectively improved and the development cost can be reduced. In addition, the processing logic encapsulated in the SDK corresponding to the clone tool can be flexibly updated according to the requirements, so as to facilitate adaptation to a plurality of different file cloning scenarios.

[0062] In some example embodiments, after the cloning operation of the source file is completed, the client 10 can implement the query of file metadata by using the destination file on the destination management node 202. In this implementation, the client 10 is further configured to: in response to an access request for the destination file, send a query request to the destination management node 202. After receiving the query request, the destination management node 202 can return the respective shard identifiers of the at least one destination file shard corresponding to the destination file to the client 10. It should be understood that the at least one destination file shard here refers to one or more destination file shards that need to be accessed by the query request. The query request of the client 10 can carry the offset and length information corresponding to the destination file, and the destination management node 202 can return the shard identifiers of the one or more destination file shards according to the offset and length information. The client 10 can access the corresponding storage nodes according to the respective shard identifiers of the at least one destination file shard. Accordingly, the accessed storage nodes can access the corresponding data blocks according to the respective shard identifiers of the at least one destination file shard.

[0063] Optionally, the access operation of the client 10 to the destination management node 202 can be implemented by a cloning tool, which can provide an open(dstfilename) interface for opening the destination file, wherein the interface parameter dstfilename is used to configure the file identifier of the destination file. The client 10 can send a query request to the destination management node 202 by calling the open(dstfilename) interface, and obtain the shard identifiers of the at least one file shard returned by the management node 202. Then, the client 10 can access the data blocks from the corresponding storage nodes according to the shard identifiers of the at least one file shard.

[0064] In this implementation, after the destination management node 202 completes the cloning of the source file, the same services as the source management node 201 can be provided, thereby realizing the collaborative services of multiple federations and improving the service efficiency of the management node.

[0065] In some example embodiments, the cloned destination file supports an append write operation of the client 10, which can generate a new data block. In such an implementation, after the client 10 clones the destination file as a source file, the client 10 is further configured to, in response to an append write request for the destination file, send a write file request to the destination management node 202 according to the to-be-written data corresponding to the append write request. After receiving the write file request, the destination management node 202 allocates a first data block in a target storage node for the to-be-written data in the storage cluster 30, and generates a shard identifier of a first destination file shard corresponding to the first data block. The first data block can include one or more data blocks, which can be determined according to the size of the to-be-written data. The append write request can carry the size of the to-be-written data.

[0066] The destination management node 202 can return the identifier information of the target storage node allocated for the to-be-written data and the location information of the first data block on the target storage node to the client 10. In turn, the client 10 can communicate with the target storage node according to the identifier information of the target storage node, and write the to-be-written data into the first data block of the target storage node according to the location information of the first data block.

[0067] After allocating the target storage node and the first data block for the to-be-written data, the destination management node 202 can record the shard identifier of the first destination file shard and the location information of the first data block on the storage node in the metadata structure. After the target storage node successfully writes the first data block, the destination management node 202 can persistently save the above information recorded in the metadata structure for subsequent queries.

[0068] In some example embodiments, for any data block, the data block is deleted from the storage node only when the data block corresponds to multiple file shards that are all deleted. Otherwise, the data block is still retained. After the client 10 clones the destination file as a source file, the client 10 is further configured to, in response to a deletion request for any second destination file shard in the destination file, determine a second data block corresponding to the second destination file shard, and determine whether the second data block has a corresponding relationship with other file shards other than the second destination file shard. If the second data block does not have a corresponding relationship with other file shards, the second destination file shard and the second data block are deleted. If the second data block has a corresponding relationship with other file shards, the second destination file shard is deleted and the second data block is retained. In turn, other file shards can still access the second data block through the corresponding index node.

[0069] In some exemplary embodiments, the modification on the source file can be synchronized to the destination file, and the modification on the destination file can be synchronized to the source file. Specifically, after the source file is cloned as the destination file, the modification on the destination file can be directly performed on the data block corresponding to the destination file, and thus, when the data block is accessed through the source file, the modified result of the data block can be accessed. Correspondingly, the modification on the source file can be directly performed on the data block corresponding to the source file, and thus, when the data block is accessed through the destination file, the modified result of the data block can be accessed. Taking the modification on the destination file as an example, the client 10 is further configured to: in response to a modification request on the destination file, determine the shard identifier of any third destination file shard to be modified, and send the shard identifier of the third destination file shard and the modification request to the storage node where the third destination file shard is located. Then, the storage node where the third destination file shard is located can modify the third data block according to the modification request and the correspondence between the shard identifier of the third destination file shard and the third data block. Then, when the third data block is accessed again through the source file, the modified third data block can be accessed.

[0070] In this implementation, when the destination file is modified, the modified result of the destination file can be synchronized to the source file without synchronous modification on the source file, so that the throughput performance of the source management node 201 is no longer a bottleneck for the horizontal expansion of the management node, and the freedom degree of the horizontal expansion of the management node is improved.

[0071] The following will be described in detail with reference to the signaling sequence diagram. Figure 2 The signaling sequence diagram further exemplarily describes the interaction operation of the distributed file system 100 when performing file cloning. As shown in the signaling sequence diagram, Figure 2 The file cloning operation mainly includes:

[0072] Step 1. The client can call the clone preparation interface provided by the SDK to initiate the file cloning operation. In the interface parameter of the clone preparation interface, the file name of the source file to be cloned and the file of the specified destination file are carried.

[0073] Step 2. The SDK responds to the call operation of the clone preparation interface, and requests the source management node to obtain the shard identifier of the source file shard of the source file.

[0074] Step 3. The source management node returns the shard identifier of the source file shard according to the request of the SDK.

[0075] Step 4. The SDK requests the destination management node to create a temporary destination file and open the destination file.

[0076] Step 5. The destination management node returns a response message of creating and opening the destination file.

[0077] Step 6. The SDK generates an identification list of the destination file fragments of the destination management node, which is generated according to the fragment identification of the source file fragments.

[0078] Step 7. The SDK returns the identification list of the destination file fragments of the destination management node to the client.

[0079] Step 8. The client requests the storage node cluster to establish a hard link between the data blocks corresponding to the destination file fragments and the source file fragments.

[0080] Step 9. The storage node cluster processes the hard link request, which is specifically described in the foregoing embodiments.

[0081] Step 10. The storage node cluster returns a hard link response result.

[0082] Step 11. The client calls the clone end interface of the SDK.

[0083] Step 12. The SDK can send a clone submission instruction or a clone termination instruction to the destination management node. If the hard link response result returned by the storage node cluster is that the hard link is successfully established, the SDK can send the clone submission instruction. If the hard link response result returned by the storage node cluster is that the hard link fails to be established, the SDK can send the clone termination instruction. The clone submission instruction can include the identification list of the destination file fragments obtained in step 6.

[0084] Step 13. If the clone submission instruction is received, the destination management node performs a clone submission operation: saves the correspondence between the fragment identification of the destination file fragment and the destination file, modifies the temporary file name of the destination file to the formal file name, and publishes the destination file.

[0085] Step 14. If the clone termination instruction is received, the destination management node performs a clone termination operation: deletes the temporary destination file.

[0086] Step 15. The destination management node returns a corresponding response message to the SDK according to the execution result of step 13 or step 14.

[0087] Step 16. The SDK returns a response message to the client according to the received response message.

[0088] If the response message received by the client is a submission response message, the client can take the destination file as a cloned file of the source file.

[0089] In addition to the distributed file system described in the foregoing embodiments, the embodiments of the present application also provide a file cloning method of a distributed file system, which will be described below with reference to the accompanying drawings.

[0090] Figure 3Fig. 1 is a flowchart of a file cloning method provided by an example embodiment of the present application, which can include the steps shown in Fig. 1 when executed on the client side. Figure 3

[0091] Step 301, in response to a cloning request for a source file in a source management node, obtaining respective shard identifiers of a plurality of source file shards corresponding to the source file from the source management node; the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored in a plurality of storage nodes in a storage cluster.

[0092] Step 302, sending a file creation request to a destination management node, the file creation request being used to make the destination management node create a destination file.

[0093] Step 303, generating respective shard identifiers of a plurality of destination file shards according to the shard identifiers of the plurality of source file shards.

[0094] Step 304, sending a hard link establishment request to the storage cluster according to the respective shard identifiers of the plurality of source file shards and the respective shard identifiers of the plurality of destination file shards, the hard link establishment request being used to make the storage cluster establish a correspondence between the respective shard identifiers of the plurality of destination file shards and the plurality of data blocks according to a correspondence between the plurality of source file shards and the plurality of data blocks.

[0095] Step 305, when the storage cluster returns a hard link establishment success message, sending a cloning submission instruction to the destination management node according to the respective shard identifiers of the plurality of destination file shards, the cloning submission instruction being used to make the destination management node establish an association between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file.

[0096] Step 306, after receiving a submission response message returned by the destination management node, taking the destination file as a cloned file of the source file.

[0097] In some example embodiments, after taking the destination file as the cloned file of the source file, the method further includes: in response to an access request for the destination file, sending a query request to the destination management node, the query request being used to make the destination management node return respective shard identifiers of at least one destination file shard corresponding to the destination file; and sending an access request to a corresponding storage node according to the respective shard identifiers of the at least one destination file shard, the access request being used to make the corresponding storage node access a corresponding data block according to a correspondence between the respective shard identifiers of the at least one destination file shard and the data block.

[0098] ​In some example embodiments, after the destination file is cloned from the source file, the method further comprises: in response to an append write request for the destination file, sending a write file request to the destination management node according to data to be written corresponding to the append write request, the write file request being used to cause the destination management node to allocate a first data block in a target storage node for the data to be written and generate a shard identifier of a first destination file shard corresponding to the first data block in the storage cluster; receiving identifier information of the target storage node and location information of the first data block in the target storage node returned by the destination management node; and writing the data to be written corresponding to the append write request into the first data block in the target storage node according to the identifier information of the target storage node and the location information of the first data block in the target storage node, the first data block being different from a plurality of data blocks corresponding to the plurality of source file shards.

[0099] In some example embodiments, after the destination file is cloned from the source file, the method further comprises: in response to a deletion request for any second destination file shard in the destination file, determining a second data block corresponding to the second destination file shard; if the second data block has no corresponding relationship with other file shards other than the second destination file shard, deleting the second destination file shard and the second data block; and if the second data block has a corresponding relationship with other file shards other than the second destination file shard, deleting the second destination file shard and retaining the second data block.

[0100] In some example embodiments, the method further comprises: in response to a modification request for the destination file, determining a shard identifier of any third destination file shard to be modified; and sending the shard identifier of the third destination file shard and the modification request to a storage node where the third destination file shard is located, the modification request being used to cause the storage node to modify a third data block according to a corresponding relationship between the shard identifier of the third destination file shard and the third data block.

[0101] In some example embodiments, in response to a clone request for a source file in a source management node, obtaining respective shard identifiers of a plurality of source file shards corresponding to the source file from the source management node comprises: in response to the clone request, configuring an interface parameter of a clone preparation interface of a clone tool according to a source file identifier and a preset destination file identifier; invoking the clone preparation interface of the clone tool according to the configured interface parameter, so as to read, by the clone tool, a file corresponding to the source file identifier from the source management node as the source file to be cloned and obtain respective shard identifiers of the plurality of source file shards corresponding to the source file; and sending a file creation request to a destination management node, comprising: sending, by the clone tool, a file creation request to the destination management node according to the destination file identifier, so that the destination management node creates the destination file according to the destination file identifier.

[0102] In the present embodiment, when cloning a source file on a source management node, a client can obtain shard identifiers of a plurality of source file shards corresponding to the source file from the source management node, and generate respective shard identifiers of a plurality of destination file shards according to the shard identifiers of the plurality of source file shards. Based on the shard identifiers of the source file shards and the shard identifiers of the destination file shards, the client can send a hard link establishment request to a storage cluster, so that the storage cluster can establish a correspondence between the shard identifiers of the destination file shards and data blocks according to the correspondence between the source file shards and the data blocks. The client can request a destination management node to create a destination file, and notify the destination management node to establish an association between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file. Further, an access operation on the destination file on the destination management node can be linked to data blocks corresponding to the source file, so that cloning of the source file is realized without copying and migrating the data blocks, and the complexity of file cloning is reduced.

[0103] Figure 4 FIG. 1 is a flowchart of a file cloning method according to another example embodiment of the present application. The method can be executed on a source management node and can comprise the steps shown in FIG. 1. Figure 4

[0104] Step 401: receiving a file creation request sent by a client.

[0105] ​Step 402, creating a destination file and returning a creation success message to the client, the creation success message being used to make the client generate the shard identifiers of the plurality of destination file shards according to the shard identifiers of the plurality of source file shards and send a hard link establishment request to the storage cluster according to the shard identifiers of the plurality of source file shards and the shard identifiers of the plurality of destination file shards; the plurality of source file shards belong to a source file on a source management node, the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in the storage cluster; the hard link request is used to trigger the storage cluster to establish a correspondence between the shard identifiers of the plurality of destination file shards and the plurality of data blocks according to the correspondence between the plurality of source file shards and the plurality of data blocks.

[0106] Step 403, in response to the clone submission instruction sent by the client according to the shard identifiers of the plurality of destination file shards, establishing an association between the destination file and the shard identifiers of the plurality of destination file shards and publishing the destination file.

[0107] Step 404, returning a submission response message to the client, the submission response message being used to make the client take the destination file as a cloned file of the source file.

[0108] In this embodiment, the destination management node can create a destination file at the request of the client. After the client generates the shard identifiers of the plurality of destination file shards according to the shard identifiers of the plurality of source file shards and requests the storage cluster to establish a mapping relationship between the shard identifiers of the plurality of destination file shards and the data blocks corresponding to the shard identifiers of the plurality of source file shards, the destination management node can establish an association between the destination file and the shard identifiers of the plurality of destination file shards and publish the destination file. Further, an access operation on the destination file on the destination management node can be linked to the data blocks corresponding to the source file, so that the cloning of the source file is realized without the need for copying and migrating the data blocks, thereby reducing the complexity of file cloning.

[0109] Figure 5 is a flowchart of a file cloning method provided by another exemplary embodiment of the present application, which can include the steps shown in Figure 5 when executed on the storage node side.

[0110] Step 501, receiving a hard link establishment request sent by a client, the hard link request being sent by the client after creating a destination file on a destination management node.

[0111] Step 502, according to the hard link establishment request, obtaining the respective shard identifiers of the at least one source file shard and the respective shard identifiers of the at least one destination file shard, the respective shard identifiers of the at least one destination file shard being generated according to the shard identifiers of the at least one source file shard; the at least one source file shard belongs to a source file on a source management node, and the at least one source file shard corresponds to at least one data block on a storage node.

[0112] Step 503, according to the correspondence between the at least one source file shard and the at least one data block, establishing the correspondence between the respective shard identifiers of the at least one destination file shard and the respective data blocks corresponding to the at least one source file shard.

[0113] Step 504, sending a hard link establishment success message to the client, the hard link establishment success message being used to make the client send a clone submission instruction to a destination management node according to the respective shard identifiers of a plurality of destination file shards after receiving the hard link establishment success messages returned by a plurality of storage nodes in a storage cluster, the clone submission instruction being used to make the destination management node establish an association between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file.

[0114] In some exemplary embodiments, one way of establishing the correspondence between the respective shard identifiers of the at least one destination file shard and the respective data blocks corresponding to the at least one source file shard according to the correspondence between the at least one source file shard and the at least one data block can include: for any destination file shard in the at least one destination file shard, determining a source file shard corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one destination file shard; determining a target data block corresponding to the source file shard as a target data block corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one data block; creating a first inode according to the metadata information of the target data block, the metadata information at least including address information of the target data block on a storage node; establishing the correspondence between the first inode and the shard identifier of the destination file shard.

[0115] In some example embodiments, the way of establishing the correspondence between the shard identifier of each of the at least one destination file shard and the data block corresponding to the at least one source file shard according to the correspondence between the at least one source file shard and the at least one data block can include: for any of the at least one destination file shard, determining the source file shard corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one destination file shard; if the source file shard is not a cloned file shard, determining a second index node corresponding to the source file shard, the second index node being used to record address information of the target data block corresponding to the source file shard on the storage node; and establishing the correspondence between the second index node and the shard identifier of the destination file shard.

[0116] In this implementation, any storage node in the storage cluster can establish the correspondence between the shard identifier of the destination file shard and the data block according to the correspondence between the source file shard and the data block when receiving the hard link establishment request of the client. Based on the correspondence, the destination file and the data block on the storage cluster can be connected through the hard link, and then the access operation on the destination file shard can be linked to the data block corresponding to the source file, thereby realizing the cloning of the source file. In this implementation, the data in the data block does not need to be copied and migrated, and the low-complexity file cloning can be realized without affecting the consistency and integrity of the data.

[0117] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 301 to 304 can be device A; for another example, the execution subject of steps 301 and 302 can be device A, and the execution subject of step 303 can be device B; and the like.

[0118] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a certain order are included, but it should be clear that these operations can be executed in the order appearing in this text or in parallel, and the serial numbers of the operations, such as 301, 302, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order, nor do "first" and "second" represent different types.

[0119] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0120] Figure 6 The structure schematic diagram of the electronic device provided by an example embodiment of the present application is illustrated, as shown in the figure, the electronic device includes a memory 601 and a processor 602. Figure 6

[0121] The memory 601 is used for storing computer programs and can be configured to store other various data to support the operation on the electronic device. Examples of these data include instructions for any application program or method operating on the electronic device.

[0122] In some optional embodiments, Figure 6 As shown in the figure, the electronic device runs a client in a distributed file system, the processor 602 is coupled with the memory 601 and is used for executing the computer program in the memory 601, so as to: in response to a clone request for a source file in a source management node, obtain the respective shard identifiers of a plurality of source file shards corresponding to the source file from the source management node; the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in a storage cluster; send a file creation request to a destination management node, the file creation request being used to make the destination management node create a destination file; generate a plurality of respective shard identifiers of a plurality of destination file shards according to the respective shard identifiers of the plurality of source file shards; according to the respective shard identifiers of the plurality of source file shards and the respective shard identifiers of the plurality of destination file shards, send a hard link establishment request to the storage cluster, the hard link establishment request being used to make the storage cluster establish a corresponding relationship between the respective shard identifiers of the plurality of destination file shards and the plurality of data blocks according to the corresponding relationship between the plurality of source file shards and the plurality of data blocks; when the storage cluster returns a hard link establishment success message, send a clone submission instruction to the destination management node according to the respective shard identifiers of the plurality of destination file shards, the clone submission instruction being used to make the destination management node establish an association relationship between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file; after receiving a submission response message returned by the destination management node, take the destination file as a clone file of the source file.

[0123] ​Optionally, the processor 602 is further configured to, after cloning the destination file from the source file, send a query request to the destination management node in response to an access request for the destination file, the query request being used to cause the destination management node to return respective shard identifiers of at least one destination file shard corresponding to the destination file; and send an access request to a corresponding storage node according to the respective shard identifiers of the at least one destination file shard, the access request being used to cause the corresponding storage node to access a corresponding data block according to a correspondence between the respective shard identifiers of the at least one destination file shard and the data block.

[0124] Optionally, the processor 602 is further configured to, after cloning the destination file from the source file, send a write file request to the destination management node in response to an append write request for the destination file, the write file request being used to cause the destination management node to allocate a first data block in a target storage node for the to-be-written data corresponding to the append write request and generate a shard identifier of a first destination file shard corresponding to the first data block; receive, from the destination management node, identification information of the target storage node and location information of the first data block in the target storage node; and write the to-be-written data corresponding to the append write request to the first data block in the target storage node according to the identification information of the target storage node and the location information of the first data block in the target storage node, the first data block being different from a plurality of data blocks corresponding to the plurality of source file shards.

[0125] Optionally, the processor 602 is further configured to, after cloning the destination file from the source file, in response to a deletion request for any second destination file shard in the destination file, determine a second data block corresponding to the second destination file shard; if the second data block has no correspondence with other file shards except the second destination file shard, delete the second destination file shard and the second data block; and if the second data block has a correspondence with other file shards except the second destination file shard, delete the second destination file shard and retain the second data block.

[0126] Optionally, the processor 602 is further configured to, in response to a modification request for the destination file, determine a shard identifier of any third destination file shard to be modified; and send the shard identifier of the third destination file shard and the modification request to a storage node where the third destination file shard is located, the modification request being used to cause the storage node to modify a third data block according to a correspondence between the shard identifier of the third destination file shard and the third data block.

[0127] Optionally, the processor 602, in response to a clone request for a source file in a source management node, acquires the respective shard identifiers of the plurality of source file shards corresponding to the source file from the source management node, is specifically configured to: in response to the clone request, configure interface parameters of a clone preparation interface of a clone tool according to a source file identifier and a preset destination file identifier; call the clone preparation interface of the clone tool according to the configured interface parameters, so as to read, by using the clone tool, a file corresponding to the source file identifier from the source management node as the source file to be cloned and acquire the respective shard identifiers of the plurality of source file shards corresponding to the source file; and the processor 602, in sending a file creation request to a destination management node, is specifically configured to: send, by using the clone tool, a file creation request to the destination management node according to the destination file identifier, so that the destination management node creates the destination file according to the destination file identifier.

[0128] In some other optional embodiments, Figure 6 The electronic device shown can be implemented as a destination management node in a distributed file system. The processor 602 is coupled to the memory 601 and configured to execute a computer program in the memory 601 to: receive a file creation request sent by a client; create a destination file and return a creation success message to the client, the creation success message being used to make the client generate respective shard identifiers of a plurality of destination file shards according to respective shard identifiers of a plurality of source file shards and send a hard link establishment request to the storage cluster according to the respective shard identifiers of the plurality of source file shards and the respective shard identifiers of the plurality of destination file shards; the plurality of source file shards belong to a source file on a source management node, the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in the storage cluster; the hard link request is used to trigger the storage cluster to establish a correspondence between the respective shard identifiers of the plurality of destination file shards and the plurality of data blocks according to the correspondence between the plurality of source file shards and the plurality of data blocks; in response to a clone submission instruction sent by the client according to the respective shard identifiers of the plurality of destination file shards, establish an association between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file; and return a submission response message to the client, the submission response message being used to make the client take the destination file as a cloned file of the source file.

[0129] In some other optional embodiments, Figure 6The electronic device shown can be implemented as any storage node in a distributed file system, and the processor 602 is coupled with the memory 601 and used to execute a computer program in the memory 601 for: receiving a hard link establishment request sent by a client, the hard link request being sent by the client after creating a target file on a target management node; obtaining, according to the hard link establishment request, at least one source file shard and at least one target file shard, respectively, and the shard identifier of the at least one target file shard being generated according to the shard identifier of the at least one source file shard; the at least one source file shard belonging to a source file on a source management node, and the at least one source file shard corresponding to at least one data block on a storage node; establishing, according to the correspondence between the at least one source file shard and the at least one data block, a correspondence between the shard identifier of the at least one target file shard and the data block corresponding to the at least one source file shard, respectively; and sending a hard link establishment success message to the client, the hard link establishment success message being used to make the client send a clone submission instruction to the target management node according to the shard identifier of the plurality of target file shards after receiving the hard link establishment success message returned by a plurality of storage nodes in a storage cluster, and the clone submission instruction being used to make the target management node establish an association between the target file and the shard identifier of the plurality of target file shards and publish the target file.

[0130] Optionally, when the processor 602 establishes, according to the correspondence between the at least one source file shard and the at least one data block, a correspondence between the shard identifier of the at least one target file shard and the data block corresponding to the at least one source file shard, respectively, the processor 602 is specifically used for: for any target file shard in the at least one target file shard, determining a source file shard corresponding to the target file shard according to the correspondence between the at least one source file shard and the at least one target file shard; determining a target data block corresponding to the source file shard as a target data block corresponding to the target file shard according to the correspondence between the at least one source file shard and the at least one data block; creating a first index node according to the metadata information of the target data block, the metadata information at least including address information of the target data block on a storage node; and establishing a correspondence between the first index node and the shard identifier of the target file shard.

[0131] Optionally, the processor 602 is configured to: for any one of the at least one source file fragment, determine a source file fragment corresponding to the at least one source file fragment according to the correspondence between the at least one source file fragment and the at least one data block; if the source file fragment is not a cloned file fragment, determine a second index node corresponding to the source file fragment, the second index node being used to record address information of a target data block corresponding to the source file fragment on a storage node; and establish a correspondence between the second index node and a fragment identifier of the at least one destination file fragment.

[0132] Further, as shown in Figure 6 , the electronic device further includes a communication component 603, a power supply component 604, a display component 605, an audio component 606, and other components. Figure 6 Some components are only schematically shown in the electronic device, and it does not mean that the electronic device only includes Figure 6 the components shown. Figure 6 In the embodiment, the components in the dashed box are optional components, rather than mandatory components, and the specific components can be determined according to the product form of the electronic device. The electronic device of the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, or an IOT device, or a server device such as a conventional electronic device, a cloud electronic device, or an electronic device array. If the electronic device of the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, etc., it can include Figure 6 the components in the dashed box; if the electronic device of the embodiment is implemented as a server device such as a conventional electronic device, a cloud electronic device, or an electronic device array, it can not include Figure 6 the components in the dashed box.

[0133] The memory 601 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

[0134] The communication component 603 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA)), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can be implemented based on a Near Field Communication (NFC) technology, a Radio Frequency Identification (RFID) technology, an Infrared Data Association (IrDA) technology, an Ultra Wide Band (UWB) technology, a Bluetooth (BT) technology, and other technologies.

[0135] The power supply component 604 is configured to supply power to various components of the device where the power supply component is located. The power supply component can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0136] The display component includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect a duration and a pressure associated with the touch or slide operation.

[0137] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that receives an external audio signal when the device on which the audio component is located is in an operating mode such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker that outputs audio signals.

[0138] In the embodiment, when cloning the source file on the source management node, the client can obtain the shard identifier of the source file shard corresponding to the source file from the source management node. Then, the destination management node can create a destination file and generate a shard identifier of a destination shard corresponding to the destination file according to the shard identifier of the source file shard. Based on the shard identifier of the source file shard and the shard identifier of the destination file shard, the destination management node can send a hard link establishment request to the storage cluster. The storage cluster can establish a correspondence between the shard identifier of the destination file shard and the data block according to the correspondence between the source file shard and the data block. Based on the correspondence, the destination file and the data block on the storage cluster can be connected through the hard link, and then the access operation on the destination file can be linked to the data block corresponding to the source file, thereby realizing the cloning of the overall function of the source file. In this implementation, without copying and migrating the data in the data block, the low-complexity file cloning can be realized without affecting the consistency and integrity of the data.

[0139] Correspondingly, the embodiment of the present application also provides a computer readable storage medium storing a computer program, the computer program is executed to realize each step that can be executed by the electronic device in the method embodiment.

[0140] The embodiment of the present application also provides a computer program product, comprising: computer program / instructions, the computer program / instructions are executed by the processor to realize the steps in the method provided by the embodiment of the present application.

[0141] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0142] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0143] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1

[0145] In one typical configuration, the computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0146] ​​​​​​Memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as read only memory (ROM) or flash memory, in computer readable media. Memory is an example of computer readable media.

[0147] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital video disc (DVD), or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0149] The above description is only some embodiments of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A file cloning method of a distributed file system, suitable for a client, characterized in that, The method comprises: in response to a clone request for a source file in a source management node, obtaining respective shard identifiers of a plurality of source file shards corresponding to the source file from the source management node; the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in a storage cluster; sending a file creation request to a destination management node, the file creation request being used to make the destination management node create a destination file; generating respective shard identifiers of a plurality of destination file shards according to the shard identifiers of the plurality of source file shards; sending a hard link establishment request to the storage cluster according to the respective shard identifiers of the plurality of source file shards and the respective shard identifiers of the plurality of destination file shards, the hard link establishment request being used to make the storage cluster establish a correspondence between the respective shard identifiers of the plurality of destination file shards and the plurality of data blocks according to the correspondence between the plurality of source file shards and the plurality of data blocks; when the storage cluster returns a hard link establishment success message, sending a clone commit instruction to the destination management node according to the respective shard identifiers of the plurality of destination file shards, the clone commit instruction being used to make the destination management node establish an association between the destination file and the respective shard identifiers of the plurality of destination file shards and publish the destination file; after receiving a commit response message returned by the destination management node, taking the destination file as a clone file of the source file.

2. The method of claim 1, wherein, After taking the destination file as the clone file of the source file, the method further comprises: in response to an append write request for the destination file, sending a write file request to the destination management node according to to-be-written data corresponding to the append write request, the write file request being used to make the destination management node allocate a first data block in a target storage node for the to-be-written data and generate a shard identifier of a first destination file shard corresponding to the first data block in the target storage node; receiving identifier information of the target storage node and location information of the first data block in the target storage node returned by the destination management node; writing the to-be-written data corresponding to the append write request into the first data block in the target storage node according to the identifier information of the target storage node and the location information of the first data block in the target storage node, the first data block being different from a plurality of data blocks corresponding to the plurality of source file shards.

3. The method of claim 1, wherein, After taking the destination file as the clone file of the source file, the method further comprises: in response to a deletion request for any second destination file shard in the destination file, determining a second data block corresponding to the second destination file shard; if the second data block does not have a corresponding relationship with other file shards except the second destination file shard, deleting the second destination file shard and the second data block; if the second data block has a corresponding relationship with other file shards except the second destination file shard, deleting the second destination file shard and retaining the second data block.

4. The method of claim 1, wherein, The method further comprises: In response to a modification request for the target file, determining a shard identifier of any third target file shard to be modified; sending the shard identifier of the third target file shard and the modification request to a storage node where the third target file shard is located, the modification request being used to cause the storage node to modify a third data block according to a correspondence between the shard identifier of the third target file shard and the third data block.

5. The method according to any one of claims 1-4, characterized in that, In response to a clone request for a source file in a source management node, obtaining shard identifiers of a plurality of source file shards corresponding to the source file from the source management node, comprising: In response to the clone request, configuring interface parameters of a clone preparation interface of a clone tool according to a source file identifier and a preset target file identifier; calling the clone preparation interface of the clone tool according to the configured interface parameters, so as to read a file corresponding to the source file identifier from the source management node as the source file to be cloned and obtain the shard identifiers of the plurality of source file shards corresponding to the source file; sending a file creation request to a target management node, comprising: sending a file creation request to a target management node according to the target file identifier by using the clone tool, the file creation request being used to cause the target management node to create the target file according to the target file identifier.

6. The method according to any one of claims 1-4, characterized in that, After the target file is taken as the cloned file of the source file, further comprising: In response to an access request for the target file, sending a query request to the target management node, the query request being used to cause the target management node to return shard identifiers of at least one target file shard corresponding to the target file; sending an access request to a corresponding storage node according to the shard identifiers of the at least one target file shard, the access request being used to cause the corresponding storage node to access a corresponding data block according to a correspondence between the shard identifiers of the at least one target file shard and the data block.

7. A file cloning method of a distributed file system, applicable to a destination management node, characterized in that, comprising: receiving a file creation request sent by a client; creating a target file and returning a creation success message to the client, the creation success message being used to cause the client to generate shard identifiers of a plurality of target file shards according to shard identifiers of a plurality of source file shards and send a hard link establishment request to a storage cluster according to the shard identifiers of the plurality of source file shards and the shard identifiers of the plurality of target file shards; the plurality of source file shards belong to a source file on a source management node, the plurality of source file shards correspond to a plurality of data blocks, and the plurality of data blocks are stored on a plurality of storage nodes in the storage cluster; the hard link request is used to trigger the storage cluster to establish a correspondence between the plurality of target file shards and the plurality of data blocks according to a correspondence between the plurality of source file shards and the plurality of data blocks; in response to a clone submission instruction sent by the client according to the shard identifiers of the plurality of target file shards, establishing an association relationship between the target file and the shard identifiers of the plurality of target file shards and publishing the target file; returning a commit response message to the client, the commit response message being used to cause the client to treat the destination file as a clone file of the source file.

8. A file cloning method of a distributed file system, applicable to any storage node in a storage cluster, characterized in that, comprising: receiving a hard link establishment request sent by a client, the hard link request being sent by the client after the client creates a destination file on a destination management node; according to the hard link establishment request, obtaining shard identifiers of at least one source file shard and shard identifiers of at least one destination file shard, the shard identifiers of the at least one destination file shard being generated according to shard identifiers of the at least one source file shard; the at least one source file shard belonging to a source file on a source management node, the at least one source file shard corresponding to at least one data block on a storage node; according to a correspondence between the at least one source file shard and the at least one data block, establishing a correspondence between the shard identifiers of the at least one destination file shard and the data blocks corresponding to the at least one source file shard respectively; sending a hard link establishment success message to the client, the hard link establishment success message being used to cause the client to send a clone commit instruction to the destination management node according to the shard identifiers of the plurality of destination file shards after receiving the hard link establishment success messages returned by a plurality of storage nodes in a storage cluster, the clone commit instruction being used to cause the destination management node to establish an association between the destination file and the shard identifiers of the plurality of destination file shards and publish the destination file.

9. The method of claim 8, wherein, according to a correspondence between the at least one source file shard and the at least one data block, establishing a correspondence between the shard identifiers of the at least one destination file shard and the data blocks corresponding to the at least one source file shard respectively, comprising: for any destination file shard in the at least one destination file shard, determining a source file shard corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one destination file shard; determining a target data block corresponding to the source file shard as a target data block corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one data block; creating a first inode according to metadata information of the target data block, the metadata information at least including address information of the target data block on a storage node; establishing a correspondence between the first inode and the shard identifier of the destination file shard.

10. The method according to claim 8 or 9, characterized in that, according to a correspondence between the at least one source file shard and the at least one data block, establishing a correspondence between the shard identifiers of the at least one destination file shard and the data blocks corresponding to the at least one source file shard respectively, comprising: for any destination file shard in the at least one destination file shard, determining a source file shard corresponding to the destination file shard according to the correspondence between the at least one source file shard and the at least one destination file shard; If the source file chunk is not a cloned file chunk, a second index node corresponding to the source file chunk is determined, the second index node being used to record address information of a target data block corresponding to the source file chunk on a storage node; A corresponding relationship between the second index node and a chunk identifier of the destination file chunk is established.

11. An electronic device, comprising: Comprise: a memory and a processor; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to perform the steps in the method of any one of claims 1-10.

12. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the file cloning method of any one of claims 1-10.

13. A computer program product, characterised in that, Comprise: computer programs / instructions, which are executed by the processor to implement the file cloning method of any one of claims 1-10.

Images