A metadata management method and system adapted to distributed file system
By using consistent hashing algorithms and subdirectory sharding technology in distributed file systems, the problem of unbalanced metadata distribution is solved, load balancing and efficient read and write operations are achieved, and the performance of distributed file systems is improved.
Patent Information
- Application Number
- CN202411109881.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-14
AI Technical Summary
In a distributed file system, when multiple MDS servers are used for metadata storage and management, the number of metadata distribution is unbalanced due to sharding in units of directories, which makes it difficult to optimize the read and write speed and generates a lot of communication overhead.
The consistent hashing algorithm is used to disperse file metadata evenly on multiple MDS servers, and subdirectory sharding and file metadata replication are performed when needed to ensure that each MDS server is load-balanced and reduce network communication overhead.
It realizes load balancing in metadata storage process, reduces communication overhead, improves read and write speed, and improves the performance of distributed file systems.
Smart Images

Figure CN118643019B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a metadata management method and system suitable for a distributed file system. Background Art
[0002] In a distributed file system, an MDS server (Metadata Server) is used to store and manage metadata. The metadata includes the file's attributes, location, access location, etc., which helps the distributed file system manage the files themselves and plays an important role in improving the user's access efficiency and operation efficiency of the distributed file system.
[0003] In an actual distributed file system, in order to improve the reading and writing speed of metadata to improve the performance of the distributed file system, it is usually implemented in the following ways: Method 1 is: from the perspective of memory capacity, a buffer elimination algorithm is introduced to keep a certain number of hot metadata in the memory at all times to reduce the occupation of memory capacity, and finally the metadata reading and writing speed is improved in the case of a large amount of metadata by improving the content performance. However, a single MDS server is used in this type of approach. At this time, the MDS server not only needs to process the reading and writing of metadata, but also needs to process the buffer elimination algorithm; and these will lead to limited metadata reading and writing performance. In order to make up for the defects of method 1, method 2 is generated to optimize the metadata reading and writing speed. Specifically, the method 2 is: introducing multiple MDS servers to share the load of a single MDS metadata in metadata storage and management. In order to evenly distribute metadata to multiple MDS servers and ensure that the load of each MDS is relatively balanced; the general approach is: first, the entire directory tree is sharded in units of directories, and then the Hash algorithm is introduced to take the modulus of the number of MDS servers to shard the directory to different MDS servers.
[0004] However, the inventors found that although sharding by directory has technical advantages in facilitating local management of MDS servers, due to the different number of specific files in each directory, such methods will inevitably cause an imbalance in the number of metadata managed by multiple MDSs. In order to alleviate this problem, a large amount of metadata in each MDS server needs to be migrated, resulting in a large amount of communication overhead. This will go against the purpose of using multiple MDS servers in the first place, and ultimately there will be a technical defect that the read and write speed is difficult to further optimize when storing and managing metadata. Summary of the invention
[0005] The purpose of the present invention is to provide a metadata management method and system suitable for a distributed file system, so as to improve the technical problem of uneven distribution of metadata quantity when sharding is performed in units of directories during the current process of metadata storage and management on multiple MDS servers.
[0006] To achieve the above object, the present invention proposes the following technical solutions:
[0007] In the first aspect, the technical solution provides a metadata management method adapted to a distributed file system, including:
[0008] Obtain metadata of all files to be stored and managed to construct a file set, and obtain metadata of each directory to which each file metadata belongs to construct a directory set;
[0009] The directory set is stored in the first MDS server, and the file set is processed based on a consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server; and at the same time, a communication connection is established between the first MDS server and each second MDS server;
[0010] Send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; then enable the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
[0011] Further, the directory set is stored in the first MDS server, and the file set is processed based on a consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server; and at the same time, a communication connection is established between the first MDS server and each second MDS server; including:
[0012] When it is determined that the number of bytes of the directory set is greater than a preset byte number threshold, metadata of each sub-directory is extracted from the first MDS server and stored in a plurality of third MDS servers;
[0013] At this time, a communication connection is established between the first MDS server and each third MDS server, and a communication connection is established between each third MDS server and each second MDS server.
[0014] Further, the sending of the file read and write request to the first MDS server to obtain the target directory metadata, and continuing to parse and obtain the target file metadata contained in the target directory metadata, and the plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; includes:
[0015] When it is determined that there are multiple file read / write requests, and they come from different clients and request the same file, extract the target file metadata corresponding to the file read / write request from the corresponding target second MDS server, and copy the target file metadata to the remaining second MDS servers;
[0016] Among them, the read and write operations of each client on the same file are not exactly the same;
[0017] Obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively;
[0018] When it is determined that each client has received a correct feedback result of the file read / write request, the target file metadata copied in the remaining second MDS servers is deleted.
[0019] Further, the sending of the file read and write request to the first MDS server to obtain the target directory metadata, and continuing to parse and obtain the target file metadata contained in the target directory metadata, and the plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; includes:
[0020] When it is determined that the file read / write request has a format abnormality, a format modification instruction is sent to the client; wherein the format modification instruction includes the abnormal location;
[0021] When it is determined that there is content abnormality in the file read and write request, a first alarm instruction is sent to the client, and a second alarm instruction is sent to the main control end; wherein, the first alarm instruction is used to remind the user that the corresponding file metadata exceeds its read and write permissions, and the second alarm instruction includes the corresponding client IP, request timestamp and request content.
[0022] Further, the sending of the file read and write request to the first MDS server to obtain the target directory metadata, and continuing to parse and obtain the target file metadata contained in the target directory metadata, and the plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; includes:
[0023] When it is determined that the preset storage update cycle has been reached and the amount of stored data in the second MDS server is less than the preset first data amount threshold, a heartbeat detection is performed on all second MDS servers based on the dynamic subtree partitioning method to re-shard the file metadata in each second MDS server so that the actual amount of stored data between any two second MDS servers is less than the preset second data amount threshold.
[0024] In the second aspect, the technical solution provides a metadata management system adapted to a distributed file system, including:
[0025] An acquisition module is used to acquire metadata of all files to be stored and managed to construct a file set, and acquire metadata of each directory to which each file metadata belongs to construct a directory set;
[0026] A storage module is used to store the directory set in the first MDS server, process the file set based on a consistent hashing algorithm to break up the metadata of each file in the file set and evenly distribute it to each second MDS server; and establish a communication connection between the first MDS server and each second MDS server;
[0027] A management module is used to send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; thereby enabling the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
[0028] Furthermore, the storage module includes:
[0029] A sub-storage unit, configured to extract metadata of each sub-directory from the first MDS server and store the metadata in a plurality of third MDS servers when it is determined that the number of bytes of the directory set is greater than a preset byte number threshold;
[0030] At this time, a communication connection is established between the first MDS server and each third MDS server, and a communication connection is established between each third MDS server and each second MDS server.
[0031] Furthermore, the management module includes:
[0032] a replication unit, configured to extract target file metadata corresponding to the file read / write request from the corresponding target second MDS server when determining that there are multiple file read / write requests and that they come from different clients and request the same file, and to replicate the target file metadata to the remaining second MDS servers; wherein the read / write operations of the same file by each client are not completely the same;
[0033] An operation unit, used to obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively;
[0034] The deleting unit is used to delete the target file metadata copied in the remaining second MDS servers when it is determined that each client has received a correct feedback result of the file read / write request.
[0035] In a third aspect, the technical solution provides an electronic device, comprising at least one processor, wherein the processor is coupled to a memory, wherein a computer program is stored in the memory, and wherein the computer program is configured to execute the method described when executed by the processor.
[0036] In a fourth aspect, the present technical solution provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the described method when executed by a computer.
[0037] Beneficial effects:
[0038] It can be seen from the above technical solutions that the present application provides a metadata management method adapted to a distributed file system to improve the technical defect of uneven distribution of the amount of managed metadata caused by sharding in units of directories during the current metadata storage process based on multiple MDS servers.
[0039] The method includes: first, obtaining metadata of all files to be stored and managed to construct a file set, and obtaining metadata of each directory to which each file metadata belongs to construct a directory set. Secondly, storing the directory set in the first MDS server, processing the file set based on a consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server; at the same time, establishing a communication connection between the first MDS server and each second MDS server. Finally, sending a file read and write request to the first MDS server to obtain the target directory metadata, and continuing to parse and obtain the target file metadata contained in the target directory metadata, and the several target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
[0040] It can be seen that the present application has designed a new solution for metadata storage. Considering that most file systems are files rather than directories, it is considered to manage the metadata of the directory and the metadata of the file separately. Specifically, the distribution of directory metadata adopts a strategy different from the distribution of file metadata. That is, all directory metadata are uniformly managed by a single first MDS server, while the metadata of the file is still distributed. Specifically, in order to facilitate the partitioned storage of file metadata and the uniformity of the load of each second MDS server after storage, a consistent hashing algorithm is introduced to directly break up the metadata of the file and evenly distribute it to different second MDS servers.
[0041] Furthermore, when the method is used to check all files in a directory, any directory request will be sent from the first MDS server to other second MDS servers. Except for the first MDS server needing to communicate with other second MDS servers, other second MDS servers do not need to communicate with each other. This also greatly reduces network communication overhead.
[0042] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered to be part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent.
[0043] The foregoing and other aspects, embodiments and features of the present invention can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned from the practice of the specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in every figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which:
[0045] Figure 1 This is a flowchart of the metadata management method adapted to a distributed file system described in this embodiment;
[0046] Figure 2 A flowchart of another method for storing directory metadata;
[0047] Figure 3 This is a processing flow chart when there are multiple file read and write requests;
[0048] Figure 4 A flowchart for determining when a file read or write request is abnormal;
[0049] Figure 5 A flowchart for performing storage update in a second MDS server;
[0050] Figure 6 This is a structural block diagram of the metadata management system adapted to the distributed file system described in this embodiment;
[0051] Figure 7 is a structural block diagram of the electronic device described in this embodiment. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meaning understood by people with general skills in the field to which the present invention belongs.
[0053] The words "first", "second" and similar words used in the specification and claims of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular form of "a", "an" or "the" and other similar words do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and the like mean that the elements or objects appearing before "include" or "comprise" cover the features, wholes, steps, operations, elements and / or components listed after "include" or "comprise", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0054] In an actual distributed file system, when sharding is performed in units of directories to achieve uniformly distributed storage of metadata on multiple MDS servers, although it facilitates the local management of the MDS server, due to the different number of specific files under each directory, such a method is bound to cause an imbalance in the number of metadata managed by multiple MDS servers; and in order to alleviate this problem, a large amount of metadata in each MDS server needs to be migrated, resulting in a large amount of communication overhead. Ultimately, it is difficult to further optimize the read and write speed when performing metadata storage and management. Based on this, this embodiment aims to provide a metadata management method adapted to a distributed file system to improve the above technical defects.
[0055] The following is a detailed introduction to the metadata management method for a distributed file system described in this embodiment in conjunction with the accompanying drawings.
[0056] like Figure 1 As shown, the method includes:
[0057] Step S102: Obtain metadata of all files to be stored and managed to construct a file set, and obtain metadata of each directory to which each file metadata belongs to construct a directory set.
[0058] In this embodiment, the file metadata includes the file name, size, type, access rights, etc. The target metadata is queried and operated by means of an index and index structure based on the target metadata.
[0059] Step S104: store the directory set in the first MDS server, process the file set based on the consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server; and establish a communication connection between the first MDS server and each second MDS server.
[0060] As a preferred implementation, in order to improve the logic of directory metadata storage and thus improve the reading and writing speed, Figure 2 As shown, the following steps are also included:
[0061] Step S10402: when it is determined that the number of bytes of the directory set is greater than a preset byte number threshold, metadata of each sub-directory is extracted from the first MDS server and stored in a third MDS server.
[0062] At this time, a communication connection is established between the first MDS server and each third MDS server, and a communication connection is established between each third MDS server and each second MDS server. Therefore, based on step S10402, when there are too many target metadata, the directory metadata and the sub-target metadata can be stored in different MDS servers, thereby improving the logic of the index structure, and finally achieving the effect of improving the indexing efficiency and the metadata reading and writing speed.
[0063] Step S106: Send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; and then enable the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
[0064] Considering that in the actual file access process, there are multiple clients that request to read and write a file at the same time. At this time, all requests will fall on the same or several second MDS servers. The corresponding second MDS servers need to process these requests in sequence, which is inefficient. Based on this, combined with Figure 3 As shown, this embodiment improves this problem in the following ways:
[0065] Step S10602: When it is determined that there are multiple file read and write requests and they come from different clients and request the same file, extract the target file metadata corresponding to the file read and write requests from the corresponding target second MDS server, and copy the target file metadata to the remaining second MDS servers.
[0066] The read and write operations of the same file by each client are not completely the same. Specifically, if the number of file read and write requests is N, the target file metadata copied to the remaining second MDS servers is N-1 copies.
[0067] Step S10604: obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively.
[0068] In this embodiment, the read and write operations include: create, delete, move, rename, etc. In specific implementation, if the file read and write request sent by client A is the second in order of timestamp, the second MDS server storing the metadata of the first copied target file responds to its request.
[0069] Step S10606: When it is determined that each client has received a correct feedback result of the file read / write request, the target file metadata copied in the remaining second MDS servers is deleted.
[0070] At this time, based on steps S10602 to S10606, it can be avoided that all file read and write requests fall on the same or several second MDS servers, thereby increasing the load pressure of the single MDS server and thereby improving the read and write speed.
[0071] Further, combined with Figure 4 As shown, exceptions during the reading and writing process are also handled in the following ways:
[0072] Step S10622: When it is determined that the file read / write request has a format abnormality, a format modification instruction is sent to the client.
[0073] In a specific implementation, the format modification instruction includes an exception location.
[0074] Step S10624: When it is determined that the file read / write request has abnormal content, a first alarm instruction is sent to the client, and a second alarm instruction is sent to the master control terminal.
[0075] In a specific implementation, the first alarm instruction is used to remind the user that the corresponding file metadata exceeds its read and write permissions, and the second alarm instruction includes the corresponding client IP, request timestamp and request content.
[0076] At this time, based on steps S10622 to S10624, the file read and write request exceptions can be classified and processed, thereby improving the read and write effectiveness and efficiency while avoiding data leakage.
[0077] As a further preferred implementation, considering that the deletion of some file metadata may cause an imbalance in the amount of data in each second MDS server, Figure 5 As shown, the following steps are also included:
[0078] Step S108: When it is determined that the preset storage update cycle has been reached and the amount of stored data in the second MDS server is less than the preset first data amount threshold, a heartbeat detection is performed on all second MDS servers based on the dynamic subtree partitioning method to re-shard the file metadata in each second MDS server so that the actual amount of stored data of any two second MDS servers is less than the preset second data amount threshold.
[0079] At this time, based on step S108, a dynamic subtree partitioning algorithm is introduced to perform compensation distribution in each second MDS server to ensure that the data volume in each second MDS server is always in a basically balanced state.
[0080] Meanwhile, when there is new file metadata to be stored, steps S102 to S108 are repeated.
[0081] In summary, the method described in this embodiment designs a new management method, that is, storing directories and files in different ways, avoiding the defect of introducing a large amount of communication overhead while ensuring load balancing of each MDS server. It effectively improves the reading and writing speed, thereby improving the performance of the distributed file system.
[0082] The above program can be run in the processor, or it can be stored in the memory (or computer-readable storage medium), which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.
[0083] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.
[0084] Based on this, this embodiment also provides a metadata management system adapted to a distributed file system. Figure 6 As shown, the system comprises:
[0085] The acquisition module is used to acquire metadata of all files to be stored and managed to construct a file set, and acquire metadata of each directory to which each file metadata belongs to construct a directory set.
[0086] A storage module is used to store the directory set in the first MDS server, process the file set based on a consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server; and at the same time establish a communication connection between the first MDS server and each second MDS server.
[0087] A management module is used to send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; thereby enabling the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
[0088] The system is used to implement the steps of the above method, so what has been explained will not be repeated here.
[0089] For example, the storage module further includes:
[0090] A sub-storage unit, configured to extract metadata of each sub-directory from the first MDS server and store the metadata in a plurality of third MDS servers when it is determined that the number of bytes of the directory set is greater than a preset byte number threshold;
[0091] At this time, a communication connection is established between the first MDS server and each third MDS server, and a communication connection is established between each third MDS server and each second MDS server.
[0092] For another example, the management module includes:
[0093] A replication unit is used to determine that there are multiple file read and write requests, and they come from different clients, and when requests are made for the same file, extract the target file metadata corresponding to the file read and write requests from the corresponding target second MDS server, and replicate the target file metadata to the remaining second MDS servers; wherein the read and write operations of each client on the same file are not exactly the same.
[0094] The operation unit is used to obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively.
[0095] The deleting unit is used to delete the target file metadata copied in the remaining second MDS servers when it is determined that each client has received a correct feedback result of the file read / write request.
[0096] This embodiment also provides an electronic device. Figure 7 As shown, it includes at least one processor, the processor is coupled to a memory, a computer program is stored in the memory, and the computer program is configured to execute the above method when executed by the processor.
[0097] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the above method is implemented.
[0098] Since the system, electronic device and storage medium are all built based on the method and used to execute the method, the reading and writing speed of metadata can be effectively improved in actual application, thereby improving the performance of the distributed file system.
[0099] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.
Claims
1. A metadata management method adapted to a distributed file system, characterized in that: include: Obtain metadata of all files to be stored and managed to construct a file set, and obtain metadata of each directory to which each file metadata belongs to construct a directory set; The directory set is stored in the first MDS server, and the file set is processed based on a consistent hashing algorithm to break up the metadata of each file in the directory set and evenly distribute the metadata to each second MDS server; At the same time, a communication connection is established between the first MDS server and each second MDS server; wherein, it also includes: when it is determined that the number of bytes of the directory set is greater than a preset byte number threshold, each sub-directory metadata is extracted from the first MDS server and stored in a plurality of third MDS servers; at this time, a communication connection is established between the first MDS server and each third MDS server, and a communication connection between each third MDS server and each second MDS server; Send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; then enable the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
2. The metadata management method adapted to a distributed file system according to claim 1, characterized in that: The sending of a file read / write request to the first MDS server to obtain target directory metadata, and continuing to parse and obtain target file metadata contained in the target directory metadata, and a plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; comprises: When it is determined that there are multiple file read / write requests, and they come from different clients and request the same file, extract the target file metadata corresponding to the file read / write request from the corresponding target second MDS server, and copy the target file metadata to the remaining second MDS servers; Among them, the read and write operations of each client on the same file are not exactly the same; Obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively; When it is determined that each client has received a correct feedback result of the file read / write request, the target file metadata copied in the remaining second MDS servers is deleted.
3. The metadata management method adapted to a distributed file system according to claim 1, characterized in that: The sending of a file read / write request to the first MDS server to obtain target directory metadata, and continuing to parse and obtain target file metadata contained in the target directory metadata, and a plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; comprises: When it is determined that the file read / write request has a format abnormality, a format modification instruction is sent to the client; wherein the format modification instruction includes the abnormal location; When it is determined that there is content abnormality in the file read and write request, a first alarm instruction is sent to the client, and a second alarm instruction is sent to the main control end; wherein, the first alarm instruction is used to remind the user that the corresponding file metadata exceeds its read and write permissions, and the second alarm instruction includes the corresponding client IP, request timestamp and request content.
4. The metadata management method adapted to a distributed file system according to claim 1, characterized in that: The sending of a file read / write request to the first MDS server to obtain target directory metadata, and continuing to parse and obtain target file metadata contained in the target directory metadata, and a plurality of target second MDS servers where the target file metadata is located; and then enabling the first MDS server to communicate with the plurality of target second MDS servers simultaneously to read and write the target file metadata; comprises: When it is determined that the preset storage update cycle has been reached and the amount of stored data in the second MDS server is less than the preset first data amount threshold, a heartbeat detection is performed on all second MDS servers based on the dynamic subtree partitioning method to re-shard the file metadata in each second MDS server so that the actual amount of stored data on any second MDS server is less than the preset second data amount threshold.
5. A metadata management system adapted to a distributed file system, characterized in that: include: An acquisition module is used to acquire metadata of all files to be stored and managed to construct a file set, and acquire metadata of each directory to which each file metadata belongs to construct a directory set; A storage module, used for storing the directory set in the first MDS server, processing the file set based on a consistent hashing algorithm to break up the metadata of each file therein and evenly distribute it to each second MDS server, and establishing a communication connection between the first MDS server and each second MDS server; The storage module further includes: a sub-storage unit, which is used to extract metadata of each sub-directory from the first MDS server and store it in a plurality of third MDS servers when it is determined that the number of bytes of the directory set is greater than a preset byte number threshold; at this time, a communication connection between the first MDS server and each third MDS server, and a communication connection between each third MDS server and each second MDS server are correspondingly established; A management module is used to send a file read and write request to the first MDS server to obtain the target directory metadata, and continue to parse and obtain the target file metadata contained in the target directory metadata, and several target second MDS servers where the target file metadata is located; thereby enabling the first MDS server to communicate with the several target second MDS servers at the same time to read and write the target file metadata.
6. The metadata management system adapted to a distributed file system according to claim 5, characterized in that: The management module comprises: a replication unit, configured to extract target file metadata corresponding to the file read / write request from the corresponding target second MDS server when determining that there are multiple file read / write requests and that they come from different clients and request the same file, and to replicate the target file metadata to the remaining second MDS servers; wherein the read / write operations of the same file by each client are not completely the same; An operation unit, used to obtain the timestamp of each file read and write request and the number of each second MDS server storing the target file metadata; and match each file read and write request to the corresponding second MDS server in sequence according to the timestamp order and the number order, and then perform read and write operations on the target file metadata therein respectively; The deleting unit is used to delete the target file metadata copied in the remaining second MDS servers when it is determined that each client has received a correct feedback result of the file read / write request.
7. An electronic device, characterized in that: The method comprises at least one processor, wherein the processor is coupled to a memory, wherein a computer program is stored in the memory, and wherein the computer program is configured to execute the method according to any one of claims 1 to 4 when executed by the processor.
8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Data management method and device, computer equipment and storage medium
CN111427841A
