Distributed file storage method and apparatus

Abstract

The specification provides a distributed file storage method and device. The method is applied to a metadata node in a distributed storage system, and the system further includes a plurality of storage nodes. The method comprises: identifying directory names of each subdirectory and a file name of a target file to be stored from a logical path of the target file; calculating a first hash value of each directory name and a second hash value of the file name respectively; sequentially splicing each first hash value in a descending order of directory levels of each subdirectory, and splicing the splicing result of each first hash value and the second hash value as an index of the target file; determining a target storage node from the plurality of storage nodes according to the index, and storing the target file in the target storage node.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of terminal technology, and in particular to a distributed file storage method and apparatus. Background Technology

[0002] Distributed storage systems typically rely on distributed storage nodes to store files / data. A typical EB-scale distributed storage system can store trillions of files. For such massive amounts of data, it is necessary to distribute it across different storage nodes as much as possible to avoid overloading a single machine.

[0003] However, because the files are stored in different storage nodes, file query operations often require frequent switching between different storage nodes, resulting in low file query efficiency, which urgently needs to be improved. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a distributed file storage method is proposed, applied to a metadata node in a distributed storage system, the system further comprising multiple storage nodes, the method comprising:

[0006] For the target file to be stored, the directory names of each subdirectory and the file name of the target file are identified from the logical path of the target file;

[0007] Calculate the first hash value of each directory name and the second hash value of the file name respectively; and concatenate each first hash value in descending order of directory level of each subdirectory, and concatenate the concatenation result of each first hash value with the second hash value to form the index of the target file;

[0008] The target storage node is determined from the plurality of storage nodes according to the index, and the target file is stored in the target storage node.

[0009] According to a second aspect of one or more embodiments of this specification, a distributed file storage device is provided for use in a metadata node of a distributed storage system, the system further comprising multiple storage nodes, the device comprising:

[0010] The name recognition unit is used to identify the directory names of each subdirectory and the file name of the target file from the logical path of the target file to be stored.

[0011] The hash concatenation unit is used to calculate the first hash value of each directory name and the second hash value of the file name respectively; and to concatenate each first hash value in descending order of the directory level of each subdirectory, and to concatenate the concatenation result of each first hash value with the second hash value to form the index of the target file;

[0012] The file storage unit is used to determine the target storage node from the plurality of storage nodes according to the index, and to store the target file in the target storage node.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in the first aspect by executing the executable instructions.

[0014] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0015] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0016] As can be seen from the above embodiments, the metadata nodes in the system calculate hash values ​​based on directory names and file names to determine the index of the target file. For each subdirectory in the logical path, the first hash value of each directory name is concatenated in the hierarchical order of the subdirectories. Then, the preliminary concatenation result is further concatenated with the second hash value of the file name to obtain the index of the target file. Finally, based on the index, the target storage node for storing the target file is determined from the storage cluster, and the storage of the target file is completed.

[0017] Understandably, because this scheme concatenates the first hash value of each directory name sequentially according to the hierarchical order of subdirectories, and further concatenates the initial concatenation result with the second hash value of the file name, the concatenation order of the hash values ​​is the same as the positional order of each subdirectory and file name in the logical path. This ensures that the resulting index (i.e., the complete hash value formed by sequentially concatenating the first and second hash values) effectively and accurately preserves the local characteristics of the file. Specifically, this scheme not only maps the high-dimensional data of the file's logical path to the low-dimensional data of the index (hash value), but also ensures that files with similar logical paths have similar indexes, thereby ensuring that these files are stored in similar locations in the cluster (i.e., stored on nearby, adjacent, or even the same storage nodes); in other words, it can distribute files with similar logical paths to the same storage nodes as much as possible. Obviously, storing files with indexes generated in the above manner can minimize jumps between different storage nodes when querying files, thereby effectively improving file query efficiency. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the architecture of a distributed storage system provided in an exemplary embodiment.

[0019] Figure 2 This is a flowchart of a distributed file storage method provided in an exemplary embodiment.

[0020] Figure 3 This is a schematic diagram of a tree structure of a logical path provided in an exemplary embodiment.

[0021] Figure 4 This is a schematic diagram of a hash concatenation process provided in an exemplary embodiment.

[0022] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0023] Figure 6 This is a block diagram of a distributed file storage device provided in an exemplary embodiment. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0025] Distributed storage systems typically rely on distributed storage nodes to store files / data. A typical EB-scale distributed storage system can store trillions of files. For such massive amounts of data, it is necessary to distribute it across different storage nodes as much as possible to avoid overloading a single machine.

[0026] However, because the files are stored in different storage nodes, file query operations often require frequent switching between different storage nodes, resulting in low file query efficiency, which urgently needs to be improved.

[0027] Local order-preserving hashing is a hash function that maintains the local characteristics of the input data. Its core objective is to map high-dimensional data to a low-dimensional space while preserving the local relationships between the data. For example, it converts data in a multi-dimensional space into low-dimensional hash values, ensuring that data points that are close in the original space remain close in the hashed space.

[0028] Several schemes based on local order-preserving hashing algorithms have been proposed for file storage, but their effectiveness remains unsatisfactory. For example, the AngleCut scheme uses a local order-preserving hashing algorithm to encode keys, mapping nodes to one or more rings through depth-first traversal. Its hash value consists of two parts: a tag and an ang. The tag is a 1-bit key consisting of the ring sequence number and some reserved bits, while ang is the angle of the node's projection onto the ring. While this scheme can achieve file locality to some extent, its adaptability to dynamic scenarios such as changes in file paths or nodes is poor. Another example is the DeepHash scheme, which uses machine learning to learn the local order-preserving hash function in the AngleCut scheme. This involves employing network embedding techniques combined with query frequency, embedding metadata nodes into fixed-length feature vectors as training instances, and designing a paired loss function for training. Although this scheme improves file query performance, the hyperparameters used in the model training process are all based on the dataset, and different hyperparameters result in vastly different file query performances, and it is only applicable to specific datasets.

[0029] To address this, this solution proposes a novel distributed file storage scheme based on a local order-preserving hash algorithm: the hash value is calculated directly using the logical path of the file, and the file index is obtained by concatenating the hash values ​​in order. The file is then stored in the target storage node determined by the index, so that files with similar logical addresses are stored in the same storage node as much as possible, thereby effectively improving the file query efficiency while maintaining file locality.

[0030] The solution will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Figure 1This is a schematic diagram of the architecture of a distributed storage system provided in an exemplary embodiment. For example... Figure 1 As shown, the system includes a storage cluster composed of multiple nodes, such as cluster 11 which includes nodes 111, 112, and 113. In one scenario, the nodes in the storage cluster can be divided into master nodes and storage nodes, such as node 111 being the master node and nodes 112 and 113 being storage nodes. In another scenario, all nodes in the storage cluster can be storage nodes. In this case, the system also needs to include a management node for the storage cluster, such as nodes 111, 112, and 113 in cluster 11 being storage nodes, with server 12 outside the cluster serving as the management node for the cluster. The aforementioned master node or management node can serve as the metadata node described in the embodiments of this specification.

[0032] In some embodiments, the distributed storage system can interface with application services and provide data storage functionality to those services. For example... Figure 1 As shown, the system can consist of only cluster 11, or it can consist of cluster 11 and management node 12. The application services interfaced with the distributed storage system can include application server 13 and client 14.

[0033] Figure 1 Any of the storage node, management node, and server 13 shown can be a physical server containing an independent host, or a virtual server hosted by a host cluster; the client 14 can be any type of electronic device used by the user, such as a tablet, laptop, PDA (Personal Digital Assistant), wearable device (such as smart glasses, smartwatch, etc.), etc., and one or more embodiments in this specification do not limit this. Furthermore, Figure 1 The network 10, which enables interaction between the parties, can be based on the communication methods supported by the corresponding electronic devices. Specifically, it can be either wired or wireless network, but this specification does not impose any restrictions on this.

[0034] Figure 2 This is an exemplary embodiment of a distributed file storage method, which is applied to metadata nodes (such as...) in a distributed storage system. Figure 1 The system also includes multiple storage nodes (such as the master node 111 or the management node 12 of cluster 11 shown). Figure 1 (e.g., node 112 in cluster 11 shown). Figure 2 As shown, the method includes the following steps 202-206.

[0035] Step 202: For the target file to be stored, identify the directory names of each subdirectory and the file name of the target file from the logical path of the target file.

[0036] As the implementing entity of the distributed file storage method described in this specification, the metadata node is used to manage the metadata of files stored in the storage cluster and manage file storage and read / write operations based on the metadata. The metadata describes key information such as file attributes, location, and organizational relationships, and is used to support file management, access, and maintenance. Specifically, the metadata of the target file may include file time information (such as creation time, modification time, access time, etc.), file size, access permissions, encryption information, verification information, copy information, owner information, etc. For example, it may include the path mapping relationships described below.

[0037] In one embodiment, the target file to be stored can be generated by an application service that interfaces with the distributed storage system. In this case, the metadata node can obtain the logical path of the target file from the application service. For example, Figure 1 After generating the target file to be stored, the server 13 shown can send a file storage request for the target file to the metadata node, and then the metadata node can extract the logical path of the target file carried in the request.

[0038] In one embodiment, the logical path described in this specification can be divided into two parts: a folder and a filename. The folder consists of at least one subdirectory, and if multiple subdirectories are included, they are arranged sequentially in descending order of directory hierarchy. Furthermore, the filename can have any suffix, and this specification does not impose any limitation on this.

[0039] The logical path conforms to a tree structure, which contains at least one directory level (corresponding one-to-one with the subdirectories contained in the target). Figure 3 Taking the tree structure shown as an example, this tree structure contains four directory levels. The first level contains a folder named "User" (i.e., the first level contains a subdirectory named "User", the same applies below); the second level contains two folders named "Data" and "Type", which will not be elaborated further. Let's assume that the second-level folder named "Type" contains a file named "Document1.mp3" (file 1); the fourth-level folder named "Filemaps" contains a file named "Document2.mp4" (file 2); and the fourth-level folder named "Roaming" contains a file named "Document3.pdf" (file 3) and a file named "Document4.txt" (file 4). Taking file 3 as an example... Figure 4As shown, its logical path is "\Users\data\AppData\Roaming\Document3.pdf", the directory within this path is "\Users\data\AppData\Roaming\", and the file name is "Document3.pdf". The logical path structure of other files is similar and will not be described further.

[0040] As can be seen from the above embodiments, in the logical path of the target file, the subdirectories are arranged in descending order of directory level, with higher-level subdirectories appearing earlier in the list and lower-level subdirectories appearing later in the list. Therefore, the directory level of each subdirectory is positively correlated with its appearance in the logical path.

[0041] Of course, this specification does not limit the number of subdirectories contained in the logical path, the length of each directory name (i.e., the number of characters), or the text format (language type, etc.) used for the directory and file names.

[0042] Step 204: Calculate the first hash value of each directory name and the second hash value of the file name; and concatenate each first hash value in descending order of directory level of each subdirectory, and concatenate the concatenation result of each first hash value with the second hash value to form the index of the target file.

[0043] After identifying the directory names and file names in the logical path, their corresponding hash values ​​can be calculated and concatenated sequentially to obtain the index of the target file. The index of the target file described in this specification is the file's key. This index is used to uniquely identify the target file in the distributed storage system, serving as the basis for subsequent file storage and retrieval operations. It is worth noting that the logical path of the target file is visible at the application layer (i.e., ordinary users of the aforementioned application can operate on the file according to this path), while the index calculated in this scheme and its final physical path are only visible to the storage layer, not to the application layer; in other words, the index is only valid within the distributed storage system and invalid in the application. Therefore, the index is the key identifier maintained by the metadata node that connects the application layer and the storage layer.

[0044] In one embodiment, given that the same content often yields different hash values ​​using different hash algorithms, and that the length of a hash value can often be pre-specified, hash values ​​can be calculated according to the corresponding hash length. For example, when calculating the first and second hash values, the hash lengths corresponding to each directory name and the file name can be determined first, and then the first hash value for each directory name and the second hash value for the file name can be calculated according to the corresponding hash. For example, for each directory name, a first hash value matching the hash length corresponding to that directory name can be calculated; and a second hash value matching the hash length corresponding to the file name can be calculated. This method ensures that each calculated first and second hash value has a definite length, thereby ensuring the accuracy of the subsequent concatenation result.

[0045] In one embodiment, the hash lengths corresponding to the aforementioned directory names and file names can be preset values. For example, if the distributed storage system pre-sets an index length E for the index to be generated, the user (such as the system administrator or maintenance personnel) can also set the maximum number of directories for the index (i.e., the maximum number of subdirectories corresponding to the hash values ​​used to concatenate the index) according to the file management rules of the application services connected to the system, and set a corresponding hash length for each subdirectory. The hash length set for any subdirectory is used as the length of the first hash value generated according to the target name of that subdirectory. Alternatively, the corresponding hash length can be temporarily calculated based on the number of subdirectories in the logical path and the hierarchical level of each subdirectory, which will not be elaborated further.

[0046] In cases where there are multiple subdirectories within the logical path (i.e., the path contains multiple subdirectories), the hash length of each directory name can be the same. This simplifies the calculation logic of the first hash value, thereby shortening the overall time spent determining the first hash and generating the index, and helping to improve the overall storage efficiency of the file. Alternatively, the hash length corresponding to the directory name of each subdirectory can be positively correlated with the total number of subdirectories containing the same file within that subdirectory, where the total number of subdirectories containing the same file is the number of subdirectories contained in the parent subdirectory to which the subdirectory belongs. This ensures that the more subdirectories belonging to the same subdirectory, the longer the first hash of those subdirectories will be. Therefore, even with a large number of potentially conflicting subdirectories, it is possible to minimize the possibility of identical first hash values ​​among these subdirectories, effectively preventing hash collisions (or overflows) and improving the stability of the generated index and the entire distributed storage system.

[0047] by Figure 3 File 3 shown is used as an example of the target file. Figure 4As shown, the logical path of the file sequentially includes four subdirectories "Users", "data", "AppData" and "Roaming" in order from highest to lowest level. The corresponding first hash values thereof are Hf1, Hf2, Hf3 and Hf4 respectively, and the hash lengths of each first hash value are b1, b2, b3 and b4 respectively. Similarly, the file name included in the logical path is "Document3.pdf", the corresponding second hash value thereof is Hn, and the hash length of this hash is N. Wherein, the hash lengths of the foregoing respective first hash values may be the same, that is, b1=b2=b3=b4. Alternatively, the hash length of each first hash value may also be positively correlated with the total number of sibling subdirectories of the corresponding subdirectory, as in Figure 3 Under the tree structure shown, b1 < b2 (because the number of subdirectories at the first level is 1, while the number of second-level subdirectories under the Users folder is 2); similarly, b2 = b4 < b3 (because the number of third-level subdirectories under the Data folder is 3, and the number of fourth-level subdirectories under the AppData folder is also 2), which will not be repeated. It can be understood that the more sibling folders included in the upper-level folder to which a certain folder belongs, the longer the hash length of the folder (and these sibling folders) will be, which helps to avoid the same first hash value being calculated for multiple sibling folders under the same folder, thereby avoiding hash collision.

[0048] The above embodiments are only described with the hash length of the first hash value. In fact, the preset hash length of the second hash of the file name can also be determined in the foregoing manner, or the hash length positively correlated with the total number of files included in the same subdirectory can be determined according to said total number, and the second hash value conforming to the hash length is calculated according to the file name, which will not be repeated.

[0049] In addition, when calculating the first hash value of any directory name, the directory name and the hash length of the directory name determined in the foregoing manner may be used as input parameters of a hash algorithm to perform a hash operation, and the first hash value is determined based on the operation result. Similarly, when calculating the second hash value of the file name, the file name and the hash length of the file name determined in the foregoing manner may be used as input parameters of a hash algorithm to perform a hash operation, and the second hash value is determined based on the operation result. Wherein, if the operation result conforms to the corresponding hash length, the result is used as the corresponding hash value; otherwise, if the length of the operation result is greater than the corresponding hash length, the operation result may be cropped (for example, only a plurality of high-order bits of the operation result that conform to the corresponding hash length are retained) to obtain the corresponding hash value.

[0050] The unit of the hash length of the hash value described in this specification may be bit, Byte, etc., which is not limited in the embodiments of this specification.

[0051] In one embodiment, after calculating the first hash values and the second hash value, a plurality of methods can be used for splicing to obtain the index of the target file. For example, each first hash value and the second hash value can be sequentially spliced according to the order of corresponding subdirectories and file names in the logical path to obtain the index, and this splicing method is simple and efficient.

[0052] For another example, a blank key with a preset length (such as the aforementioned E) (that is, a Key with all values being 0) can also be determined first, then each first hash value is sequentially placed in the blank key in order from the highest bit to the lowest bit (this placement operation is equivalent to splicing each first hash value), and the second hash value is placed at the lowest bit of the blank key. At this time, if there is no vacancy between the second hash value and the last first hash value in the blank key (that is, all first hash values and the second hash value exactly fill all bits of the blank key), the key at this time is determined as the index of the target file; and if there is a vacancy between the second hash value and the last first hash value in the blank key (that is, the first hash values and the second hash value cannot fill all bits of the blank key), a preset value can be set for the vacancy (for example, all are set to 0), and the key after the setting is completed is determined as the index of the target file.

[0053] Through this method, the preset blank key can be used as an index template, and the sequential splicing of the first hash values and the second hash value is completed by sequentially filling in the hash values according to a preset order, which not only ensures that the positions of each hash value in the generated index are accurate, but also has simple splicing logic that is easy to implement, and the index generation efficiency is relatively high. In addition, this method also allows users to flexibly change the number of bits occupied by directory names or file names, which helps meet file storage requirements under a variety of tree structures.

[0054] The complete process of generating an index described in the above embodiment can be implemented by the following pseudo-code:

[0055] Key=null;

[0056] E=E-N; / / N is the number of bytes occupied by the second hash value, and the initial value of E is the total number of bytes of the index

[0057] for(int i=0;i<paths.length();i++){ / / Traverse the path for encoding

[0058] E=E–B; / / B is an array, whose elements are sequentially the hash lengths of each subdirectory, such as b1, b2, b3

[0059] If(E≤0)break; / / Stop splicing when there are no blank bytes

[0060] Key += HF(paths[i], B); / / Loop through and calculate the first hash value of each directory name and append it to Key.

[0061] }

[0062] Pad the extra bytes with '0';

[0063] Key += HF(name(), N); / / Calculate the first hash value of the filename and append it to Key.

[0064] Return Key / / The final concatenated key

[0065] As can be seen from the pseudocode above, the input parameters of the code are the logical path (which can also be regarded as the directory names of each subdirectory and the file name of the target file), the expected hash bit length E of the index, the number of bytes N occupied by the hashed file name, the hash function HF(), and the number of bytes B (in array form) occupied by each level of subdirectory; the output result is the final calculated hash value Key (i.e., the index).

[0066] Step 206: Determine the target storage node from the plurality of storage nodes according to the index, and store the target file in the target storage node.

[0067] After calculating the index of the target file in the aforementioned manner, it can be used as a globally unique identifier for the file to be distributed and stored. That is, the target storage node is first determined from multiple storage nodes based on the index, and then the target file is stored in that node, thus realizing distributed storage of the file.

[0068] Understandably, because this scheme concatenates the first hash value of each directory name sequentially according to the hierarchical order of subdirectories, and further concatenates the initial concatenation result with the second hash value of the file name, the concatenation order of the hash values ​​is the same as the positional order of each subdirectory and file name in the logical path. This ensures that the resulting index (i.e., the complete hash value formed by sequentially concatenating the first and second hash values) effectively and accurately preserves the local characteristics of the file. Specifically, this scheme not only maps the high-dimensional data of the file's logical path to the low-dimensional data of the index (hash value), but also ensures that files with similar logical paths have similar indexes, thereby ensuring that these files are stored in similar locations in the cluster (i.e., stored on nearby, adjacent, or even the same storage nodes); in other words, it can distribute files with similar logical paths to the same storage nodes as much as possible. Obviously, storing files with indexes generated in the above manner can minimize jumps between different storage nodes when querying files, thereby effectively improving file query efficiency.

[0069] As can be seen, the scheme in this specification is completely different from the AngleCut and DeepHash schemes mentioned above. This scheme generates the file index based on the basic principle of local order-preserving hashing. It not only abandons complex algorithms such as model training, making the index encoding logic simpler and more efficient, but also can fully adapt to dynamic scenarios such as changes in file path or node.

[0070] For example, Figure 3 Files 3 and 4 are located in the same folder, so their logical paths have identical directories, differing only in filenames. Therefore, the indices calculated using the aforementioned method have a high similarity: the high-order bits (the bits corresponding to each first hash value) of the two indices are essentially the same, differing only in the low-order bits (the bits corresponding to the second hash value) of the filenames. Thus, files 3 and 4 are highly likely to be stored in the same storage node. Conversely, for the remaining files, the logical paths of files 2 and 3 differ slightly, while those of files 1 and 3 differ even more. Therefore, compared to the index of file 1, the index of file 2 is closer to the index of file 3. Consequently, file 2 will be stored closer to file 3, while file 4 will be stored further away from file 3. This demonstrates that the index generated by the above scheme and the file storage scheme implemented accordingly can effectively achieve local time-preserving characteristics of files.

[0071] In one embodiment, when determining the target storage node based on the index, the node hash value of each of the plurality of storage nodes can be determined first. Then, the target node hash value that matches the index can be determined from the various node hash values, and the storage node corresponding to the target node hash value can be determined as the target storage node. The node hash value of the storage node can be calculated based on information such as the node's communication address, node number, and node ID.

[0072] A consistent hashing algorithm can be used to determine the hash value of the target node matching the index. This method ensures that data migration is minimized when the storage cluster changes (e.g., nodes join or leave the cluster). For example, when the index length is E (e.g., 256 bits), the hash value space of 2^E can be abstracted into a virtual ring structure (with values ​​from 0 to 2^E-1), called a hash ring. Then, each storage node is mapped to its corresponding position on the hash ring based on its node hash value. Optionally, one or more virtual nodes can be generated for each storage node to address the problem of uneven data distribution on the hash ring (e.g., in a storage cluster with 10 storage nodes, 3 virtual nodes can be generated for each storage node, resulting in a total of 30 virtual nodes). Based on this, the hash value of the target file (i.e., the aforementioned index) can be mapped to its corresponding position on the hash ring, and the first virtual node can be found in a preset direction (e.g., clockwise), with the storage node corresponding to this virtual node being determined as the target storage node for the file. After determining the target storage node and storing the file using this method, if any storage node fails (such as crashing), you only need to migrate the data stored on it to the previous node. This results in a small amount of data migration and high cluster stability.

[0073] Alternatively, the target storage node can be determined using the hash modulo method. For example, if it is determined that the storage is on M storage nodes, each storage node can be numbered sequentially (e.g., 0 to M-1), and the value of Key%M can be calculated. Then, the storage node with the same number as this value can be used as the target storage node. This method is logically simple, efficient, and provides good data storage uniformity, helping to avoid excessive differences in storage node load.

[0074] In addition, in order to further improve the operating quality of the distributed storage system while maintaining the basic locality of the files, after determining a certain storage node through the aforementioned index, other storage nodes adjacent to that node (such as those in the same rack, the same server room, or the same data center) can be further determined, and a load balancing algorithm can be used to further determine the target storage node from that node and the other nodes to store the target file. This will not be elaborated further.

[0075] In one embodiment, when storing a target file in a target storage node, the physical path of the target file in the target storage node can be determined first based on the index, and then the target file can be stored according to the physical path. For example, to facilitate management and improve performance, the system may design a complex directory hierarchy for the storage node, such as creating subdirectories based on the first few characters of the file key to reduce the number of files in a single directory. The stored filename may be directly converted from its key (e.g., directly using the filename in the logical path as the stored filename), or additional information (such as version number, timestamp, copy number, etc.) may be appended to form the final storage name. Furthermore, even on the same node, there may be multiple disks or other types of storage media. Therefore, when determining the specific storage path, it is also necessary to consider how to reasonably distribute the load among these storage media, which will not be elaborated further. This method can avoid storing a large number of files in the same file, and the file path distribution effect is good.

[0076] In one embodiment, the metadata node can also maintain path mapping relationships between the logical path, the index, and the physical path, such as a first path mapping relationship between the logical path and the index, and a second path mapping relationship between the index and the physical path. Based on this, after the target file is stored, a query service for the target file can be provided to relevant parties. For example, in response to a file retrieval request initiated by a requester (such as the client or server of the aforementioned application) for the target file (e.g., a user can initiate such a request through an operation in the application), the corresponding physical path can be queried from the path mapping relationship according to the logical path carried in the file retrieval request. For example, the index of the file to be queried can be queried in the first path mapping relationship according to the logical path, and then the corresponding physical path can be queried in the second path mapping relationship according to the index. It is understood that the above physical path can indicate which node the target file is stored in (the highest-level subdirectory of the path can be the node ID of the target storage node), and can also indicate in detail which path (i.e., which folder) the target file is stored in within that node. Furthermore, the target file can be read from the target storage node according to the queried physical path, and the read target file can be returned to the requester.

[0077] In one embodiment, the logical path of the target file may change. For example, the application user may change the file's location (e.g., cut / move the file to another folder, or rename at least one folder on its logical path) or rename the file. To ensure that the saved target file can still be accurately retrieved after the path change, the metadata node can respond to the change in the target file's logical path by generating a new index according to the new logical path (the specific generation method can be found in the embodiment of steps 202-204, which will not be repeated here), and determining the new physical path corresponding to the new index; then, it updates the path mapping relationship according to the new logical path, the new index, and the new physical path; and, if the physical path differs from the new physical path, it moves the target file stored according to the physical path to the new physical path, i.e., re-stores the target file according to the new physical path. After this, the requester can initiate a file retrieval request for the file, and the metadata node can respond to the request by querying the corresponding new physical path in the new path mapping relationship based on the new logical path carried in the request, then reading the moved target file from the corresponding storage node and returning it to the requester.

[0078] This method allows for the rapid naming of new files, and the corresponding index can be quickly updated to reflect the new path when a file is moved to a different directory. For example, when a logical path changes, the above index generation scheme can quickly generate a new index, enabling rapid and accurate changes to file paths.

[0079] In one embodiment, given that this solution is applied to a distributed storage system comprising multiple storage nodes, to improve file storage stability and prevent file query errors or even loss due to target storage node failure, copies of the target file can be stored. For example, at least one file copy can be generated for the target file, and then the storage nodes corresponding to each file copy are determined based on a preset copy placement strategy, with each file stored in its corresponding storage node. The target file itself can be referred to as the primary / leader replica, and the at least one file copy can be referred to as secondary replicas or standby replicas. The primary replica and multiple secondary replicas together constitute the replica set of the target file and share the aforementioned index. However, each replica can be distinguished by a valid replica identifier within the system, such as adding keywords like "_Replica1" or "_Replica2" to the replica filename. However, for application layer users, the target file is simply a single file, and they are unaware of the multiple replicas.

[0080] Of course, the physical paths corresponding to each file replica can also be added to the aforementioned path mapping relationship (the logical paths and indexes of each file replica are the same, but the physical paths are different). Based on this, when a file retrieval request for the target file is made in the future, the actual replica to be read can be determined according to logic such as load balancing, number of reads, and file / node activity. The target file can then be read from the corresponding location according to the physical path of the replica recorded in the path mapping relationship and returned to the requester.

[0081] Figure 4 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 4As shown, device 500 mainly consists of a communication interface 502, a user interface 504, a processor 506, and a data storage 508. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 510. The communication interface 502 enables device 500 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 502 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 502 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 502 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 502 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0082] User interface 504 includes receiving user input and providing output to the user. Therefore, user interface 504 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 504 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 504 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 500 may support remote access from other devices via communication interface 502 or another physical interface (not shown). User interface 504 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 504 may also be configured as a display device for rendering or displaying text fragments.

[0083] Processor 506 may contain one or more general-purpose processors and / or special-purpose processors.

[0084] Data storage 508 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 506. Data storage 508 may include removable and non-removable components.

[0085] Processor 506 is capable of executing program instructions 518 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 508 to perform the various functions described herein. Data storage 508 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 500, enable device 500 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 518 by processor 506 may result in processor 506 using data 512.

[0086] For example, program instructions 518 may include an operating system 522 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 500 and one or more applications 520 (e.g., a browser, social application, or game application). Similarly, data 512 may include operating system data 516 and application data 514. Operating system data 516 is primarily accessible to the operating system 522, while application data 514 is primarily accessible to one or more applications 520. Application data 514 may reside in a file system visible or hidden from the user of device 500.

[0087] Application 520 can communicate with operating system 522 through one or more application programming interfaces (APIs). These APIs help application 520 read and / or write application data 514, transmit or receive information via communication interface 502, receive or display information on user interface 504, etc.

[0088] In some terminology, application 520 may be simply referred to as "app (application)". Furthermore, application 520 can be downloaded to device 500 through one or more online app stores or app markets. However, applications can also be installed on device 500 in other ways, such as through a web browser or a physical interface on device 500 (e.g., a USB port).

[0089] Please refer to Figure 6 Distributed file storage devices can be applied to, for example, Figure 5 The device shown implements the technical solution of this specification. The distributed file storage device is used in a metadata node of a distributed storage system, which also includes multiple storage nodes. The device includes:

[0090] The name recognition unit 601 is used to identify the directory names of each subdirectory and the file name of the target file from the logical path of the target file to be stored.

[0091] The hash concatenation unit 602 is used to calculate the first hash value of each directory name and the second hash value of the file name respectively; and to concatenate each first hash value in descending order of the directory level of each subdirectory, and to concatenate the concatenation result of each first hash value with the second hash value to form the index of the target file;

[0092] File storage unit 603 is used to determine a target storage node from the plurality of storage nodes according to the index, and to store the target file in the target storage node.

[0093] Optionally, the hash concatenation unit 602 is specifically used for:

[0094] Determine the hash length corresponding to each directory name and the file name;

[0095] For each directory name, calculate a first hash value that matches the hash length corresponding to that directory name; and calculate a second hash value that matches the hash length corresponding to that file name.

[0096] Optionally, the number of subdirectories in the logical path is multiple, wherein the hash length corresponding to the directory name of each subdirectory is positively correlated with the total number of subdirectories of the same file in that subdirectory, and the total number of subdirectories of the same file is the number of subdirectories contained in the parent subdirectory to which that subdirectory belongs.

[0097] Optionally, the hash concatenation unit 602 is specifically used for:

[0098] Each first hash value is placed in the blank key in order from the most significant bit to the least significant bit, and the second hash value is placed in the least significant bit of the blank key;

[0099] If there is no empty space between the second hash value and the last first hash value in the blank key, then the key at this time is determined as the index of the target file;

[0100] If there is a gap between the second hash value and the last first hash value in the blank key, a preset value is set for the gap, and the key after setting is determined as the index of the target file.

[0101] Optionally, the file storage unit 603 is specifically used for:

[0102] Determine the node hash value of each of the plurality of storage nodes, determine the target node hash value that matches the index from the node hash values, and determine the storage node corresponding to the target node hash value as the target storage node.

[0103] Optionally, the file storage unit 603 is specifically used for:

[0104] The physical path of the target file in the target storage node is determined according to the index, and the target file is stored according to the physical path.

[0105] Optionally, a path mapping unit 604 is also included, for:

[0106] Maintain the path mapping relationship between the logical path, the index, and the physical path;

[0107] After the target file is stored, in response to the file retrieval request initiated by the requester for the target file, the corresponding physical path is queried from the path mapping relationship according to the logical path carried in the file retrieval request, and the target file is read from the target storage node according to the queried physical path, and the read target file is returned to the requester.

[0108] Optionally, a path update unit 605 is also included, for:

[0109] In response to a change in the logical path of the target file, a new index is generated according to the new logical path, and the new physical path corresponding to the new index is determined.

[0110] The path mapping relationship is updated according to the new logical path, the new index, and the new physical path; and, if the physical path is different from the new physical path, the target file stored according to the physical path is moved to the new physical path.

[0111] Optionally, it also includes a replica storage unit 605 for:

[0112] At least one file copy is generated for the target file, and the storage node corresponding to each file copy is determined based on a preset copy placement strategy. Each file is then stored in its corresponding storage node.

[0113] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0114] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0115] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0116] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0117] What those skilled in the art will understand is:

[0118] In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0119] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0120] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0121] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0122] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0123] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0124] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A distributed file storage method, applied to a metadata node in a distributed storage system, the system further comprising multiple storage nodes, the method comprising: For the target file to be stored, identify the directory names of each subdirectory and the file name of the target file from the logical path of the target file; Calculate the first hash value of each directory name and the second hash value of each file name; Furthermore, the first hash values ​​are concatenated sequentially according to the directory level of each subdirectory from high to low, and the concatenation result of the first hash values ​​is concatenated with the second hash value to form the index of the target file; The target storage node is determined from the plurality of storage nodes according to the index, and the target file is stored in the target storage node.

2. The method according to claim 1, wherein calculating the first hash value of each directory name and the second hash value of the file name respectively comprises: Determine the hash length corresponding to each directory name and the file name; For each directory name, calculate the first hash value that matches the hash length corresponding to that directory name; In addition, calculate a second hash value that matches the hash length corresponding to the filename.

3. The method according to claim 1, wherein the number of subdirectories in the logical path is multiple, wherein the hash length corresponding to the directory name of each subdirectory is positively correlated with the total number of subdirectories of the same file in that subdirectory, and the total number of subdirectories of the same file is the number of subdirectories contained in the parent subdirectory to which that subdirectory belongs.

4. The method according to claim 1, wherein the step of sequentially concatenating each first hash value according to the directory level of each subdirectory from high to low, and concatenating the concatenation result of each first hash value with the second hash value to form the index of the target file, includes: Each first hash value is placed in the blank key in order from the most significant bit to the least significant bit, and the second hash value is placed in the least significant bit of the blank key; If there is no empty space between the second hash value and the last first hash value in the blank key, then the key at this time is determined as the index of the target file; If there is a gap between the second hash value and the last first hash value in the blank key, a preset value is set for the gap, and the key after setting is determined as the index of the target file.

5. The method according to claim 1, wherein determining the target storage node from the plurality of storage nodes based on the index comprises: Determine the node hash value of each of the plurality of storage nodes, determine the target node hash value that matches the index from the node hash values, and determine the storage node corresponding to the target node hash value as the target storage node.

6. The method according to claim 1, wherein storing the target file in the target storage node comprises: The physical path of the target file in the target storage node is determined according to the index, and the target file is stored according to the physical path.

7. The method according to claim 6, further comprising: Maintain the path mapping relationship between the logical path, the index, and the physical path; After the target file is stored, in response to the file retrieval request initiated by the requester for the target file, the corresponding physical path is queried from the path mapping relationship according to the logical path carried in the file retrieval request, and the target file is read from the target storage node according to the queried physical path, and the read target file is returned to the requester.

8. The method according to claim 7, further comprising: In response to a change in the logical path of the target file, a new index is generated according to the new logical path, and the new physical path corresponding to the new index is determined. The path mapping relationship is updated according to the new logical path, the new index, and the new physical path; and, if the physical path is different from the new physical path, the target file stored according to the physical path is moved to the new physical path.

9. The method according to claim 1, further comprising: At least one file copy is generated for the target file, and the storage node corresponding to each file copy is determined based on a preset copy placement strategy. Each file is then stored in its corresponding storage node.

10. A distributed file storage device, applied to a metadata node in a distributed storage system, the system further comprising multiple storage nodes, the device comprising: The name recognition unit is used to identify the directory names of each subdirectory and the file name of the target file from the logical path of the target file to be stored. A hash concatenation unit is used to calculate the first hash value of each directory name and the second hash value of the file name, respectively. Furthermore, the first hash values ​​are concatenated sequentially according to the directory level of each subdirectory from high to low, and the concatenation result of the first hash values ​​is concatenated with the second hash value to form the index of the target file; The file storage unit is used to determine the target storage node from the plurality of storage nodes according to the index, and to store the target file in the target storage node.

11. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

Images