Metadata organization method and device of distributed file system

Abstract

The application belongs to the technical field of data processing, and particularly relates to a metadata organization method of a distributed file system, which comprises the following steps: in writing, generating an internal unique identifier of file content according to the file content, and generating an internal unique identifier of directory content according to the directory content; in reading, generating a unique access identifier of a file mapping block based on a file full path name, obtaining a globally unique identifier of a file index data block in the file mapping block, and obtaining the internal unique identifier of the file content according to the obtained globally unique identifier of the file index data block, and then obtaining data of the file content; generating a unique access identifier of a directory mapping block based on a directory full path name, obtaining a globally unique identifier of a directory index data block in the directory mapping block, and obtaining the internal unique identifier of the directory content according to the obtained globally unique identifier of the directory index data block, and then obtaining data of the directory content.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a metadata organization method and device of a distributed file system. BACKGROUND

[0002] In recent years, the Internet industry has developed rapidly, and emerging industries such as the Internet of Things have also been popularized. Various forms of content are rapidly increasing, and the amount of data in various industries is growing. There are a large amount of important data and files that need to be saved. In addition, tasks such as big data analysis require fast data retrieval from storage units. Therefore, a large amount of resources is needed to store and manage these data. Therefore, efficient metadata organization and management are essential. Distributed file systems have become an important way to solve the storage and management of massive data due to their inherent versatility and simplicity.

[0003] The metadata of a distributed file system is data that describes the structural characteristics of the system, such as size, state information, and access permissions, owners, and times for each file or directory, as well as mapping relationships between directories and subdirectories / subfiles, file and file data block mapping relationships, etc. Operations on a distributed file system involve operations on metadata, and most accesses in a distributed file system are also based on metadata. Therefore, it is very meaningful to propose a metadata organization method for a distributed file system.

[0004] Currently, distributed file systems are mainly divided into two ways. One is to store all metadata separately through a metadata server, and store the entire file directory structure in memory. When there are too many files and directories, it will cause excessive storage. Moreover, when retrieving file and directory information, it is necessary to obtain it from the metadata server, and a large number of accesses will cause the metadata server to form a single point of bottleneck, resulting in low data retrieval efficiency. The second is to use a hash algorithm to map the divided data blocks to the storage module through the names of files and directories, and then complete the storage and retrieval of each data block. At the same time, by storing the information of the lower-level directories or files in the data of each level of directory, a cascading relationship is formed. When retrieving file or directory information, it is necessary to find it level by level from the root path, so as to retrieve file or directory information, it is still necessary to traverse multiple levels of directories, which has a large overhead. SUMMARY

[0005] To address the aforementioned shortcomings of existing technologies, this invention proposes a metadata organization method for a distributed file system. By storing all data and metadata as data blocks on storage nodes, large data items are broken down into smaller blocks. This aims to reduce the single-point bottleneck of centralized metadata storage. Furthermore, file and directory information is retrieved using hashes of the full paths of directories and files as unique identifiers. Compared to some distributed file systems that store information about lower-level directories or files in each directory index data block to form a cascading relationship, this method avoids the high overhead caused by multi-level directory traversal when retrieving files and their corresponding directory information.

[0006] This invention provides a method for organizing metadata in a distributed file system, the method comprising:

[0007] During writing, an internal unique identifier for the file content is generated based on the file content. The internal unique identifier of the file content and the relevant file attribute information maintained by the file system are stored in the file index data block. A semantically meaningless global unique identifier is generated for the file index data block. The global unique identifier of the file index data block is written into the directory content of the directory where the file is located. The global unique identifier of the file index data block and some attribute information of the file maintained by the file system are stored in the file mapping block. Based on the full path name of the file, a unique access identifier for the file mapping block is generated.

[0008] Generate an internal unique identifier for the directory content based on the directory content, store the internal unique identifier of the directory content and the relevant directory attribute information maintained by the file system into the directory index data block, generate a semantically meaningless global unique identifier for the directory index data block, write the global unique identifier of the directory index data block into the directory content of the current directory, store the global unique identifier of the directory index data block and some attribute information of the directory maintained by the file system into the directory mapping block, and generate a unique access identifier for the directory mapping block based on the full path name of the directory.

[0009] During reading, the unique access identifier of the file mapping block generated based on the full file path name is used to obtain the globally unique identifier of the file index data block in the file mapping block. Based on the globally unique identifier of the file index data block, the internal unique identifier of the file content is obtained, and then the data of the file content is obtained.

[0010] Based on the unique access identifier of the directory mapping block generated by the full path name of the directory, obtain the globally unique identifier of the directory index data block in the directory mapping block, and based on the globally unique identifier of the directory index data block, obtain the internal unique identifier of the directory content, and then obtain the data of the directory content.

[0011] As one improvement to the above technical solution, the step of generating an internal unique identifier for the file content based on the file content specifically involves:

[0012] The file content is file data containing hierarchical relationships;

[0013] Step 1-1) According to the splitting rules, the file data is split into multiple data blocks, and each data block has a unique identifier; among them, the data content in each data block is hashed to obtain the unique identifier corresponding to each data block;

[0014] Steps 1-2) use TLV format for encoding, combining the unique identifiers and capacities corresponding to multiple data blocks into a new data block;

[0015] If the capacity of the new data block is less than or equal to the preset maximum capacity of the data block, then skip to step 1-5.

[0016] If the capacity of the new data block is greater than the preset maximum capacity of the data block, then skip to steps 1-4.

[0017] Steps 1-4) Divide the new data block into multiple sub-data blocks according to the partitioning rules. Each sub-data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of the multiple sub-data blocks are combined into a new sub-data block. If the capacity of the new sub-data block is less than or equal to the maximum capacity of the data block set in advance, then skip to step 1-5).

[0018] If the capacity of the new sub-data block is greater than the maximum capacity of the preset data block, the new sub-data block is split again according to the splitting rules, and steps 1-4 are repeated.

[0019] Steps 1-5) The new data block is hashed using its data content to generate an internal unique identifier for the file content.

[0020] As one improvement to the above technical solution, the step of generating an internal unique identifier for the directory content based on the directory content specifically involves:

[0021] The directory content consists of multiple directory index blocks containing hierarchical relationships;

[0022] Step 2-1) Encode using TLV format, combining the unique identifiers and their capacities corresponding to multiple directory index blocks into a new directory data block;

[0023] Step 2-2) If the capacity of the new directory data block is less than or equal to the preset maximum capacity of the data block, then skip to step 2-4);

[0024] If the capacity of the new directory data block is greater than the preset maximum capacity of the data block, then skip to step 2-3.

[0025] Step 2-3) Divide the new directory data block into multiple subdirectory data blocks according to the splitting rules. Each subdirectory data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of multiple subdirectory data blocks are combined into a new subdirectory data block. If the capacity of the new subdirectory data block is less than or equal to the maximum capacity of the preset data block, then skip to step 2-4).

[0026] If the capacity of the new subdirectory data block is greater than the preset maximum capacity of the data block, then the new subdirectory data block is split again according to the splitting rules, and steps 2-3 are repeated.

[0027] Steps 2-4) The new directory data block is hashed using its data content to generate an internal unique identifier for the directory content.

[0028] As one of the improvements to the above technical solution, the file system maintains the following file attribute information: file index data block type, file index data block storage capacity, file name, file access time, file modification time, file creation time, file owner's username, file owner's group name, file permissions, unique identifier of file content, and unique identifier of the index block of the directory where the file is located.

[0029] The file system maintains some attribute information of the files, including: the file owner's username, the file's group name, and the file's permissions.

[0030] As one of the improvements to the above technical solution, the relevant directory attribute information maintained by the file system includes: directory index data block type, directory index data block storage capacity, directory name, directory access time, directory modification time, directory creation time, directory owner's username, directory group name, directory permissions, unique identifier of directory content, and unique identifier of the index block of the parent directory of the directory.

[0031] The file system maintains the following partial attribute information for directories: the username of the directory owner, the group name of the directory owner, and the permissions of the directory.

[0032] As one of the improvements to the above technical solution, the method of generating a unique access identifier for a file mapping block based on the full path name of the file is specifically: generating a unique access identifier for a file mapping block by hashing the full path name of the file.

[0033] As an improvement to the above technical solution, the method of generating a unique access identifier for a directory mapping block based on the full path name of the directory specifically involves generating a unique access identifier for the directory mapping block by performing a hash calculation on the full path name of the directory.

[0034] The present invention also provides a metadata organization apparatus for a distributed file system, the apparatus comprising:

[0035] The mapping module is used to obtain a file mapping block containing a globally unique identifier of the file index data block and partial attribute information of the file maintained by the file system, based on the full path name of the file; it is also used to obtain a directory mapping block containing a globally unique identifier of the directory index data block and partial attribute information of the directory maintained by the file system, based on the full path name of the directory.

[0036] The indexing module is used to obtain relevant attribute information of files and internal unique identifiers for storing file content; it is also used to obtain relevant attribute information of directories and internal unique identifiers for storing directory content.

[0037] The file module is used to store the mapping relationship between the internal unique identifier of the file content and the identifier of the split data block, so as to obtain the file content data;

[0038] The directory module stores the internal unique identifiers of the directory content and the mapping relationship between multiple directory index blocks in the directory, thereby retrieving the directory content data.

[0039] As an improvement to the above technical solution, the mapping module includes:

[0040] The directory mapping unit is used to obtain a directory mapping block containing a globally unique identifier of the directory index data block and partial attribute information of the directory maintained by the file system, based on the full path name of the directory; it is also used to store the globally unique identifier of the directory index data block and partial attribute information of the directory maintained by the file system; and

[0041] The file mapping unit is used to obtain a file mapping block containing a globally unique identifier of the file index data block and partial attribute information of the file maintained by the file system, based on the full path name of the file; it is also used to store the globally unique identifier of the file index data block and partial attribute information of the file maintained by the file system.

[0042] As an improvement to the above technical solution, the index module includes:

[0043] The directory index unit is used to obtain relevant directory attribute information maintained by the file system based on the globally unique identifier of the directory index data block; it is also used to generate an internal unique identifier for the directory content based on the directory content; and it is also used to store the internal unique identifier of the directory content and the relevant directory attribute information maintained by the file system; and

[0044] The file index unit is used to obtain relevant file attribute information maintained by the file system based on the globally unique identifier of the file index data block; it is also used to generate an internal unique identifier for the file content based on the file content; and it is also used to store the internal unique identifier of the file content and the relevant file attribute information maintained by the file system.

[0045] The beneficial effects of this invention compared to the prior art are:

[0046] The method of this invention segments large file data and directory contents into smaller data blocks to reduce the single-point bottleneck of centralized metadata storage. It uses hashes of the full pathnames of directories and files as unique identifiers to retrieve file and directory content data. Compared to existing distributed file systems, this method stores information about lower-level directories or files in each directory index data block, forming a cascading relationship. This avoids the overhead of multi-level directory traversal when retrieving file and directory content data. Furthermore, the use of hash digest naming for both file and directory content data ensures data integrity. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the hierarchical relationship structure of a specific embodiment of a metadata organization method for a distributed file system according to the present invention;

[0048] Figure 2 yes Figure 1 A detailed hierarchical relationship diagram of a specific embodiment of a metadata organization method for a distributed file system according to the present invention;

[0049] Figure 3 This is a schematic diagram of the metadata organization device of a distributed file system according to the present invention.

[0050] Figure label:

[0051] 201. Mapping Module 202. Index Module

[0052] 203. File Module 204. Directory Module

[0053] 2011, File Mapping Unit; 2012, Directory Mapping Unit

[0054] 2021, Document Indexing Unit; 2022, Directory Indexing Unit Detailed Implementation

[0055] The present invention will now be further described with reference to the accompanying drawings.

[0056] This invention provides a method for organizing metadata in a distributed file system, the method comprising:

[0057] During writing, an internal unique identifier for the file content is generated based on the file content; the internal unique identifier for the file content and the relevant file attribute information maintained by the file system are stored in the file index data block; a semantically meaningless global unique identifier is generated for the file index data block; the global unique identifier of the file index data block is written into the directory content of the directory where the file is located; the global unique identifier of the file index data block and some attribute information of the file maintained by the file system are stored in the file mapping block; and a unique access identifier for the file mapping block is generated based on the full path name of the file.

[0058] The specific process of generating an internal unique identifier for the file content based on the file content is as follows:

[0059] The file content is file data containing hierarchical relationships;

[0060] Step 1-1) According to the splitting rules, the file data is split into multiple data blocks, and each data block has a unique identifier; among them, the data content in each data block is hashed to obtain the unique identifier corresponding to each data block;

[0061] Steps 1-2) use TLV (Type Length Value) format for encoding, combining the unique identifiers and capacities of multiple data blocks into a new data block;

[0062] If the capacity of the new data block is less than or equal to the preset maximum capacity of the data block, then skip to step 1-5.

[0063] If the capacity of the new data block is greater than the preset maximum capacity of the data block, then skip to steps 1-4.

[0064] Steps 1-4) Divide the new data block into multiple sub-data blocks according to the partitioning rules. Each sub-data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of the multiple sub-data blocks are combined into a new sub-data block. If the capacity of the new sub-data block is less than or equal to the maximum capacity of the data block set in advance, then skip to step 1-5).

[0065] If the capacity of the new sub-data block is greater than the maximum capacity of the preset data block, the new sub-data block is split again according to the splitting rules, and steps 1-4 are repeated.

[0066] Steps 1-5) The new data block is hashed using its data content to generate an internal unique identifier for the file content.

[0067] The file attribute information maintained by the file system includes: file index data block type, file index data block storage capacity, file name, file access time, file modification time, file creation time, file owner's username, file owner's group name, file permissions, unique identifier of file content, and unique identifier of index block of the directory where the file is located.

[0068] The file system maintains some attribute information of the files, including: the file owner's username, the file's group name, and the file's permissions.

[0069] Generate an internal unique identifier for the directory content based on the directory content, store the internal unique identifier of the directory content and the relevant directory attribute information maintained by the file system into the directory index data block, generate a semantically meaningless global unique identifier for the directory index data block, write the global unique identifier of the directory index data block into the directory content of the current directory, store the global unique identifier of the directory index data block and some attribute information of the directory maintained by the file system into the directory mapping block, and generate a unique access identifier for the directory mapping block based on the full path name of the directory.

[0070] The process of generating an internal unique identifier for the directory content based on the directory content specifically involves:

[0071] The directory content consists of multiple directory index blocks containing hierarchical relationships;

[0072] Step 2-1) Encode using TLV (Type Length Value) format, combining the unique identifiers and their capacities corresponding to multiple directory index blocks into a new directory data block;

[0073] Step 2-2) If the capacity of the new directory data block is less than or equal to the preset maximum capacity of the directory data block, then skip to step 2-4);

[0074] If the capacity of the new directory data block is greater than the preset maximum capacity of the directory data block, then skip to step 2-3.

[0075] Step 2-3) Divide the new directory data block into multiple subdirectory data blocks according to the splitting rules. Each subdirectory data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of multiple subdirectory data blocks are combined into a new subdirectory data block. If the capacity of the new subdirectory data block is less than or equal to the maximum capacity of the preset data block, then skip to step 2-4).

[0076] If the capacity of the new subdirectory data block is greater than the preset maximum capacity of the data block, then the new subdirectory data block is split again according to the splitting rules, and steps 2-3 are repeated.

[0077] Steps 2-4) The new directory data block is hashed using its data content to generate an internal unique identifier for the directory content.

[0078] The relevant directory attribute information maintained by the file system includes: directory index data block type, directory index data block storage capacity, directory name, directory access time, directory modification time, directory creation time, directory owner's username, directory group name, directory permissions, unique identifier of directory content, and unique identifier of the index block of the parent directory.

[0079] The file system maintains the following partial attribute information for directories: the username of the directory owner, the group name of the directory owner, and the permissions of the directory.

[0080] Specifically, generating a unique access identifier for a file mapping block based on the full path name of the file involves generating a unique access identifier for the file mapping block by performing a hash calculation on the full path name of the file.

[0081] Specifically, generating a unique access identifier for a directory mapping block based on the full path name of the directory involves generating a unique access identifier for the directory mapping block by performing a hash calculation on the full path name of the directory.

[0082] During reading, the unique access identifier of the file mapping block generated based on the full file path name is used to obtain the globally unique identifier of the file index data block in the file mapping block. Based on the globally unique identifier of the file index data block, the internal unique identifier of the file content is obtained, and then the data of the file content is obtained.

[0083] Based on the unique access identifier of the directory mapping block generated by the full path name of the directory, obtain the globally unique identifier of the directory index data block in the directory mapping block, and based on the globally unique identifier of the directory index data block, obtain the internal unique identifier of the directory content, and then obtain the data of the directory content.

[0084] The file's internal unique identifier is used to retrieve the file's content data, and is also used for data verification and data block deduplication after the file is retrieved.

[0085] The unique internal identifier of the directory is used to retrieve the data in the directory content, and is also used for data verification and deduplication of data blocks after the directory is retrieved.

[0086] The semantically neutral globally unique identifier, which can be a UUID, is used to prevent the regeneration of upper-level multi-level directory index data blocks and directory data containing hierarchical relationships due to content modifications. It is also used to obtain index data blocks and thus retrieve partial attribute information of files and directories, as well as the internal unique identifiers of files and directories.

[0087] The unique access identifier of the file mapping block needs to have a unique correspondence with the full file path, and this unique identifier can be calculated from the full file path. This unique identifier can be obtained by hashing the full file path, and it is used to directly obtain partial attribute information of the file and the unique identifier of the index data block storing the file based on the full file path.

[0088] The unique access identifier of the directory mapping block needs to satisfy a unique correspondence with the full directory path, and the unique identifier can be calculated from the full directory path. The unique identifier can be obtained by hashing the full directory path. The unique identifier is used to directly obtain some attribute information of the directory and the unique identifier of the index data block of the storage directory based on the full directory path.

[0089] like Figure 3 As shown, the present invention also provides a metadata organization device for a distributed file system, the device comprising:

[0090] The mapping module 201 is used to obtain a file mapping block containing a globally unique identifier of a file index data block and partial attribute information of the file maintained by the file system, based on the full path name of the file; it is also used to obtain a directory mapping block containing a globally unique identifier of a directory index data block and partial attribute information of the directory maintained by the file system, based on the full path name of the directory.

[0091] The mapping module 201 includes:

[0092] The directory mapping unit 2012 is used to obtain a directory mapping block containing a globally unique identifier of the directory index data block and partial attribute information of the directory maintained by the file system, based on the full path name of the directory; it is also used to store the globally unique identifier of the directory index data block and partial attribute information of the directory maintained by the file system; and

[0093] The file mapping unit 2011 is used to obtain a file mapping block containing a globally unique identifier of the file index data block and partial attribute information of the file maintained by the file system, based on the full path name of the file; it is also used to store the globally unique identifier of the file index data block and partial attribute information of the file maintained by the file system.

[0094] Index module 202 is used to obtain relevant attribute information of files and internal unique identifiers for storing file content; it is also used to obtain relevant attribute information of directories and internal unique identifiers for storing directory content;

[0095] The index module 202 includes:

[0096] Directory index unit 2022 is used to obtain relevant directory attribute information maintained by the file system based on the globally unique identifier of the directory index data block; it is also used to generate an internal unique identifier for the directory content based on the directory content; and it is also used to store the internal unique identifier of the directory content and the relevant directory attribute information maintained by the file system; and

[0097] The file indexing unit 2021 is used to obtain relevant file attribute information maintained by the file system based on the globally unique identifier of the file index data block; it is also used to generate an internal unique identifier for the file content based on the file content; and it is also used to store the internal unique identifier of the file content and the relevant file attribute information maintained by the file system.

[0098] File module 203 is used to store the mapping relationship between the internal unique identifier of the file content and the identifiers of the segmented data blocks, thereby obtaining the file content data; and

[0099] Directory module 204 is used to store the internal unique identifier of the directory content and the mapping relationship between multiple directory index blocks in the directory, thereby obtaining the data of the directory content.

[0100] Example 1.

[0101] To better illustrate the method of this invention, assume that the full path of a directory is / home / test / conf, and that the directory contains the directories / home / test / conf / a, / home / test / conf / b, the file / home / test / conf / c.txt, and the file / home / test / conf / test.txt. Figure 1 The document describes the hierarchical relationship between the directory / home / test / conf and the directories / home / test / conf / a and / home / test / conf / b, as well as the hierarchical relationship between the directory / home / test / conf and the files / home / test / conf / c.txt and / home / test / conf / test.txt.

[0102] Each directory contains three parts: a directory mapping block, a directory index block, and directory data; each file contains three parts: a file mapping block, a file index block, and file data.

[0103] The ` / home / test / conf` directory mapping block contains a unique identifier for the ` / home / test / conf` directory index block. The ` / home / test / conf` directory index block contains an internal unique identifier for the contents of the ` / home / test / conf` directory. The ` / home / test / conf` directory data contains information about the ` / home / test / conf / a` directory index block, the ` / home / test / conf / b` directory index block, the ` / home / test / conf / c.txt` file index block, and the ` / home / test / conf / test.txt` file index block.

[0104] The ` / home / test / conf / a` directory mapping block contains a unique identifier for the ` / home / test / conf / a` directory index block. The ` / home / test / conf / a` directory index block contains an internal unique identifier for the contents of the ` / home / test / conf / a` directory. The ` / home / test / conf / a` directory data contains information about all subdirectory index blocks and subfile index blocks of ` / home / test / conf / a`.

[0105] The ` / home / test / conf / b` directory mapping block contains a unique identifier for the ` / home / test / conf / b` directory index block. The ` / home / test / conf / b` directory index block contains an internal unique identifier for the contents of the ` / home / test / conf / b` directory. The ` / home / test / conf / b` directory data contains information about all subdirectory index blocks and subfile index blocks of ` / home / test / conf / b`.

[0106] The mapping block of the file / home / test / conf / c.txt contains a unique identifier for the index block of the file / home / test / conf / c.txt, and the index block of the file / home / test / conf / c.txt contains an internal unique identifier for the content of the file / home / test / conf / c.txt.

[0107] The mapping block of the file / home / test / conf / test.txt contains a unique identifier for the index block of the file / home / test / conf / test.txt, and the index block of the file / home / test / conf / test.txt contains an internal unique identifier for the content of the file / home / test / conf / test.txt.

[0108] The following section combines the structure Figure 1 and2 This describes the process of writing data to the file test.txt in / home / test / conf, resulting in the writing of data to the file and the directory; reading data from the file test.txt in / home / test / conf, resulting in the reading of data from the file; and reading the contents of the / home / test / conf directory.

[0109] like Figure 1 and 2 As shown, the process of writing data to the file test.txt and writing data to the directory is as follows:

[0110] Step 3-1) Generate an internal unique identifier for the contents of the file / home / test / conf / test.txt based on the contents of the test.txt file. Store this internal unique identifier and the relevant file attribute information maintained by the file system into the file index block of / home / test / conf / test.txt. Generate a semantically meaningless globally unique identifier (UUID) for the file index block of / home / test / conf / test.txt. Write the globally unique identifier of the file index block into the directory data of / home / test / conf. Store the globally unique identifier of the file index block of / home / test / conf / test.txt and some file attribute information maintained by the file system into the file mapping block of / home / test / conf / test.txt. Based on the full path name of the file / home / test / conf / test.txt, generate a unique access identifier for the file mapping block.

[0111] The relevant file attribute information includes: file index data block type, file index data block storage capacity, file name, file access time, file modification time, file creation time, file owner's username, file owner's group name, file permissions, unique identifier of file content, and unique identifier of the index block of the directory where the file is located.

[0112] Some attribute information includes: the username of the file owner, the name of the group to which the file belongs, and the file permissions.

[0113] The specific process for generating an internal unique identifier for the contents of the file / home / test / conf / test.txt based on the contents of the test.txt file is as follows:

[0114] The file content is file data containing hierarchical relationships;

[0115] Step 3-1-1) According to the splitting rules, the data in the test.txt file is split into multiple data blocks, and each data block has a unique identifier; among them, the data content in each data block is hashed to obtain the unique identifier corresponding to each data block;

[0116] Step 3-1-2) Encode using TLV (Type Length Value) format, combining the unique identifiers and capacities corresponding to multiple data blocks into a new data block;

[0117] Step 3-1-3) If the capacity of the new data block is less than or equal to the preset maximum capacity of the data block, then skip to step 3-1-5);

[0118] If the capacity of the new data block is greater than the preset maximum capacity of the data block, then skip to step 3-1-4.

[0119] Step 3-1-4) Divide the new data block into multiple sub-data blocks according to the partitioning rules. Each sub-data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of the multiple sub-data blocks are combined into a new sub-data block. If the capacity of the new sub-data block is less than or equal to the maximum capacity of the data block set in advance, then skip to step 3-1-5).

[0120] If the capacity of the new sub-data block is greater than the maximum capacity of the preset data block, the new sub-data block is split again according to the splitting rules, and steps 3-1-4 are repeated.

[0121] Step 3-1-5) The new data block is hashed using its data content to generate an internal unique identifier for the contents of the file / home / test / conf / test.txt.

[0122] Specifically, generating a unique access identifier for the file mapping block based on the full path name of the file involves hashing the full path name of the test.txt file, / home / test / conf / test.txt, to generate a unique access identifier for the / home / test / conf / test.txt file mapping block.

[0123] Step 3-2) Generate an internal unique identifier for the contents of the / home / test / conf directory based on the contents of the / home / test / conf directory. Store the internal unique identifier of the / home / test / conf directory contents and the relevant directory attribute information maintained by the file system into the / home / test / conf directory index block. Generate a semantically meaningless globally unique identifier for this directory index block, which can be a UUID. Write the globally unique identifier of the directory index block into the contents of the / home / test / conf directory. Store the globally unique identifier of the / home / test / conf directory index block and some directory attribute information maintained by the file system into the / home / test / conf directory mapping block. Based on the full path name of the directory / home / test / conf, generate a unique access identifier for this directory mapping block.

[0124] The relevant directory attribute information includes: directory index data block type, directory index data block storage capacity, directory name, directory access time, directory modification time, directory creation time, directory owner's username, directory group name, directory permissions, unique identifier of directory content, and unique identifier of the index block of the parent directory.

[0125] Some attribute information includes: the username of the directory owner, the group name of the directory owner, and the permissions of the directory.

[0126] The process of generating an internal unique identifier for the directory content based on the directory content specifically involves:

[0127] The directory content consists of multiple directory index blocks containing hierarchical relationships;

[0128] Step 3-2-1) Encode using TLV (Type Length Value) format, combining the unique identifiers and their capacities corresponding to multiple directory index blocks under / home / test / conf after adding test.txt into a new directory data block;

[0129] Step 3-2-2) If the capacity of the new directory data block is less than or equal to the preset maximum capacity of the data block, then skip to step 3-2-4);

[0130] If the capacity of the new directory data block is greater than the preset maximum capacity of the data block, then skip to step 3-2-3.

[0131] Step 3-2-3) Divide the new directory data block into multiple subdirectory data blocks according to the partitioning rules. Each subdirectory data block is hashed to form a unique identifier. Then, using the TLV format, the unique identifiers and their capacities of multiple subdirectory data blocks are combined into a new subdirectory data block. If the capacity of the new subdirectory data block is less than or equal to the maximum capacity of the preset data block, then skip to step 3-2-4).

[0132] If the capacity of the new subdirectory data block is greater than the preset maximum capacity of the data block, then the new subdirectory data block is split again according to the splitting rules, and steps 3-2-3 are repeated.

[0133] Step 3-2-4) The new directory data block is hashed using its data content to generate an internal unique identifier for the directory content.

[0134] Specifically, generating a unique access identifier for the / home / test / conf directory mapping block based on the full path name of the directory / home / test / conf involves: generating a unique access identifier for the / home / test / conf directory mapping block by performing a hash calculation on the full path name of the directory / home / test / conf.

[0135] The process of reading data from file contents and directory contents is as follows:

[0136] Step 4-1) Based on the full path name of the file / home / test / conf / test.txt, generate a unique access identifier for the file mapping block of / home / test / conf / test.txt. Obtain the globally unique identifier of the file index block of / home / test / conf / test.txt within the file mapping block. Then, based on the globally unique identifier of the file index block of / home / test / conf / test.txt, obtain the internal unique identifier of the content of the file / home / test / conf / test.txt, and then obtain the data of the file content.

[0137] Specifically,

[0138] Step 4-1-1) Generate a unique access identifier for the file mapping block of / home / test / conf / test.txt by hashing the full path name of the file / home / test / conf / test.txt;

[0139] Step 4-1-2) Obtain the / home / test / conf / test.txt file mapping block using the unique access identifier of the / home / test / conf / test.txt file mapping block. From the / home / test / conf / test.txt file mapping block, obtain the permissions, owner, and group of test.txt. Check if there is read permission. If there is no read permission, return "no read permission"; if there is read permission, proceed to step 4-1-3).

[0140] Step 4-1-3) Obtain the globally unique identifier of the test.txt file index block in the / home / test / conf / test.txt file mapping block in Step 4-1-2), and use this identifier to obtain the / home / test / conf / test.txt file index block.

[0141] Step 4-1-4) Obtain a data block by using the file content identifier in the index block of the file / home / test / conf / test.txt, and read the identifiers and capacities of multiple data blocks encoded in TLV format from this data block.

[0142] Obtain the corresponding data block by identifying one of the data block identifiers. From the obtained data block, check whether the data block is a file splitting data block. If it is, proceed to step 4-1-6; otherwise, proceed to step 4-1-5.

[0143] Step 4-1-5) Obtain all corresponding data blocks using all their identifiers, and concatenate them sequentially into a new data block. From this new data block, read the identifiers and capacities of the multiple data blocks encoded in TLV format.

[0144] Next, obtain the corresponding data block through one of the data block identifiers. From the obtained data block, check whether the data block is a file splitting data block. If it is, proceed to step 4-1-6); otherwise, repeat step 4-1-5.

[0145] Step 4-1-6) Obtain all the segmented data blocks of the test.txt file by identifying the multiple data blocks, and concatenate all the segmented data blocks in order into a large data block, which is the test.txt file.

[0146] Step 4-2) Based on the full path name of the directory / home / test / conf, generate the unique access identifier of the / home / test / conf directory mapping block, the globally unique identifier of the / home / test / conf directory index data block in the / home / test / conf directory mapping block, and obtain the internal unique identifier of the / home / test / conf directory content based on the globally unique identifier of the obtained / home / test / conf directory index data block, and then obtain the data of the directory content.

[0147] Specifically,

[0148] Step 4-1-1) Generate a unique access identifier for the / home / test / conf directory mapping block by hashing the full path name of the directory / home / test / conf;

[0149] Step 4-1-2) Obtain the / home / test / conf directory mapping block using the unique access identifier of the / home / test / conf directory mapping block. Obtain the permissions, owner, and group of / home / test / conf from the / home / test / conf directory mapping block. Check if there is read permission. If there is no read permission, return no read permission; if there is read permission, proceed to step 4-1-3).

[0150] Step 4-1-3) Obtain the globally unique identifier of the / home / test / conf directory index data block in the directory mapping block in Step 4-1-2), and use this identifier to obtain the directory index data block.

[0151] Step 4-1-4) Obtain a data block by indexing the directory content in the / home / test / conf directory data block, and read the identifiers and capacities of multiple data blocks encoded in TLV format from this data block.

[0152] Retrieve the corresponding data block using one of the data block identifiers. From the retrieved data block, check if it is an index data block. If it is, proceed to step 4-1-6; otherwise, proceed to step 4-1-5.

[0153] Step 4-1-5) Obtain all corresponding data blocks using all their identifiers, and concatenate them sequentially into a new data block. From this new data block, read the identifiers and capacities of the multiple data blocks encoded in TLV format.

[0154] Retrieve the corresponding data block using one of the data block identifiers. From the retrieved data block, check if the data block is a file splitting data block. If it is, proceed to step 4-1-6; otherwise, repeat step 4-1-5.

[0155] Specifically,

[0156] Step 4-1-6) By identifying multiple data blocks, obtain the index blocks of all subdirectories and subfiles under / home / test / conf (including the index block of the / home / test / conf / a directory, the index block of the / home / test / conf / b directory, the index block of the / home / test / conf / c.txt file, and the index block of the / home / test / conf / test.txt file). From all the index blocks, you can obtain all directory information under / home / test / conf.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A metadata organization method of a distributed file system, the method comprising: when writing, generating an internal unique identifier of file content according to the file content, storing the internal unique identifier of the file content and relevant file attribute information maintained by the file system into a file index data block, generating a global unique identifier without semantics for the file index data block, writing the global unique identifier of the file index data block into directory content of a directory where the file is located, storing the global unique identifier of the file index data block and part of attribute information of the file maintained by the file system into a file mapping block, and generating a unique access identifier of the file mapping block based on a full path name of the file; generating an internal unique identifier of directory content according to the directory content, storing the internal unique identifier of the directory content and relevant directory attribute information maintained by the file system into a directory index data block, generating a global unique identifier without semantics for the directory index data block, writing the global unique identifier of the directory index data block into directory content of a current directory, storing the global unique identifier of the directory index data block and part of attribute information of the directory maintained by the file system into a directory mapping block, and generating a unique access identifier of the directory mapping block based on a full path name of the directory; when reading, generating a unique access identifier of the file mapping block based on a full path name of the file, obtaining the file mapping block through the unique access identifier, obtaining the global unique identifier of the file index data block in the file mapping block, and obtaining the file index data block according to the obtained global unique identifier of the file index data block, and further obtaining data of the file content through the internal unique identifier of the file content in the file index data block; generating a unique access identifier of the directory mapping block based on a full path name of the directory, obtaining the directory mapping block through the unique access identifier, obtaining the global unique identifier of the directory index data block in the directory mapping block, and obtaining the directory index data block according to the obtained global unique identifier of the directory index data block, and further obtaining data of the directory content through the internal unique identifier of the directory content in the directory index data block.

2. The metadata organization method of a distributed file system according to claim 1, wherein, The internal unique identifier of the file content is generated according to the file content, and specifically: The file content is file data containing a hierarchical relationship; Step 1-1) file data is divided into multiple data blocks according to a division rule, each data block having a unique identifier; wherein the data content in each data block is calculated by a hash algorithm to obtain a unique identifier corresponding to each data block; Step 1-2) a TLV format is used for encoding to combine the unique identifiers of the multiple data blocks and their capacities into a new data block; Step 1-3) if the capacity of the new data block is less than or equal to a pre-set maximum capacity of the data block, then go to step 1-5); If the capacity of the new data block is greater than the preset maximum capacity of the data block, go to step 1-4); step 1-4) re-divides the new data block into a plurality of sub-data blocks according to the splitting rule, each sub-data block is formed by hashing the data content of the sub-data block to form a unique identifier, and then a new sub-data block is formed by combining the unique identifiers and the capacities of the plurality of sub-data blocks in TLV format, if the capacity of the new sub-data block is less than or equal to the preset maximum capacity of the data block, go to step 1-5); If the capacity of the new sub-data block is greater than the preset maximum capacity of the data block, the new sub-data block is re-divided according to the splitting rule, and the step 1-4) is repeated; Step 1-5) generates the internal unique identifier of the file content by hashing the data content of the new data block.

3. The metadata organization method of a distributed file system according to claim 1, wherein, The internal unique identifier of the directory content is generated according to the directory content, and specifically: The directory content is a plurality of directory index blocks containing hierarchical relationships; Step 2-1) encodes in TLV format to combine the unique identifiers and the capacities of the plurality of directory index blocks into a new directory data block; Step 2-2) if the capacity of the new directory data block is less than or equal to the preset maximum capacity of the data block, go to step 2-4); If the capacity of the new directory data block is greater than the preset maximum capacity of the data block, go to step 2-3); Step 2-3) divides the new directory data block into a plurality of sub-directory data blocks according to the splitting rule, each sub-directory data block is formed by hashing the data content of the sub-directory data block to form a unique identifier, and then a new sub-directory data block is formed by combining the unique identifiers and the capacities of the plurality of sub-directory data blocks in TLV format, if the capacity of the new sub-directory data block is less than or equal to the preset maximum capacity of the data block, go to step 2-4); If the capacity of the new sub-directory data block is greater than the preset maximum capacity of the data block, the new sub-directory data block is re-divided according to the splitting rule, and the step 2-3) is repeated; Step 2-4) generates the internal unique identifier of the directory content by hashing the data content of the new directory data block.

4. The metadata organization method of a distributed file system according to claim 1, wherein, The relevant file attribute information maintained by the file system includes: file index data block type, file index data block storage capacity, file name, file access time, file modification time, file creation time, file owner name, file group name, file permissions, file content unique identifier, and index block unique identifier of the directory where the file is located. The part of the file attribute information maintained by the file system includes: file owner name, file group name, and file permissions.

5. The metadata organization method of a distributed file system according to claim 1, wherein, The directory attribute information maintained by the file system includes: directory index data block type, directory index data block storage capacity, directory name, directory access time, directory modification time, directory creation time, directory owner name, directory group name, directory permission, unique identifier of directory content, and index block unique identifier of the upper directory of the directory; The part of the directory attribute information maintained by the file system includes: directory owner name, directory group name, and directory permission.

6. The metadata organization method of a distributed file system according to claim 1, wherein, The unique access identifier of the file mapping block is generated based on the full path name of the file, and the unique access identifier of the directory mapping block is generated based on the full path name of the directory.

7. The metadata organization method of a distributed file system according to claim 1, wherein, The unique access identifier of the file mapping block is generated based on the full path name of the file, and the unique access identifier of the directory mapping block is generated based on the full path name of the directory.

8. A metadata organization apparatus of a distributed file system, configured to perform the method of any one of claims 1-7, characterized in that, The device comprises: a mapping module (201) configured to obtain a file mapping block with a globally unique identifier of a file index data block and part of file attribute information maintained by a file system according to a full path name of the file, and obtain a directory mapping block with a globally unique identifier of a directory index data block and part of directory attribute information maintained by the file system according to a full path name of the directory; an index module (202) configured to obtain relevant attribute information of the file and an internal unique identifier of the file content, and obtain relevant attribute information of the directory and an internal unique identifier of the directory content; a file module (203) configured to store a mapping relationship between the internal unique identifier of the file content and the data block identifier after being cut, and further obtain data of the file content; a directory module (204) configured to store a mapping relationship between the internal unique identifier of the directory content and the directory index block in the directory, and further obtain data of the directory content.

9. The metadata organization apparatus of a distributed file system according to claim 8, wherein, The mapping module (201) comprises: a directory mapping unit (2012) configured to obtain a directory mapping block with a globally unique identifier of a directory index data block and part of directory attribute information maintained by the file system according to a full path name of the directory, and store the globally unique identifier of the directory index data block and the part of directory attribute information maintained by the file system; and a file mapping unit (2011) configured to obtain a file mapping block with a globally unique identifier of a file index data block and part of file attribute information maintained by the file system according to a full path name of the file, and store the globally unique identifier of the file index data block and the part of file attribute information maintained by the file system.

10. The metadata organization apparatus of a distributed file system according to claim 8, wherein, The index module (202) comprises: a directory index unit (2022) configured to obtain relevant directory attribute information maintained by the file system according to the globally unique identifier of the directory index data block, generate an internal unique identifier of the directory content according to the directory content, store the internal unique identifier of the directory content and the relevant directory attribute information maintained by the file system, and a file index unit (2021) configured to obtain relevant file attribute information maintained by the file system according to the globally unique identifier of the file index data block, generate an internal unique identifier of the file content according to the file content, and store the internal unique identifier of the file content and the relevant file attribute information maintained by the file system. The file index unit (2021) is configured to acquire relevant file attribute information maintained by a file system according to a globally unique identifier of a file index data block, and is further configured to generate an internally unique identifier of file content according to the file content, and is further configured to store the internally unique identifier of the file content and the relevant file attribute information maintained by the file system.

Images