Data management method and device for distributed file system

Abstract

The application discloses a data management method and device of a distributed file system, and relates to the technical field of data management, and comprises the following steps: in response to a first client opening a target file in a writing mode, a target block identifier in an idle state is allocated to the target file; wherein the target block identifier is used for identifying a target data block in an object storage server; the target block identifier is returned to the first client, so that the first client writes target data into the target data block; wherein the target data comprises metadata generated when the first client performs a writing operation on the target file; the target block identifier is written into an index node of the target file, so that a second client accessing the target file accesses the target data in the target data block according to the target block identifier written in the index node. The present application can improve the access performance and data recovery speed of the distributed file system.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data management method and device for a distributed file system. Background Technology

[0002] Faced with large amounts of unstructured data, distributed file systems are needed to provide storage capacity, data access capabilities, and failover capabilities. A distributed file system mainly consists of a metadata server and an object storage server; the metadata server manages the file metadata, while the object storage server manages the file data itself.

[0003] In this architecture, when multiple clients concurrently access the same file, all metadata requests are directed to the metadata server, easily creating a single point of failure and significantly increasing I / O access latency. Simultaneously, metadata is periodically sent in batches to the metadata server for persistent storage. In the event of a failure, unpersisted metadata is lost. Recovering the lost metadata requires scanning the object storage server; the recovery speed is strongly correlated with the number and size of files, making it difficult to quickly restore data access capabilities. Therefore, under these conditions, distributed file systems suffer from poor access performance and slow business recovery after failures, impacting the stability and continuity of upper-layer applications. Summary of the Invention

[0004] This application provides a data management method and device for a distributed file system, which at least solves the problems of low access performance and slow business recovery after failure in related technologies.

[0005] This application provides a data management method for a distributed file system, applied to a metadata server, including:

[0006] In response to the first client opening the target file in write mode, a target block identifier in an idle state is allocated to the target file; wherein, the target block identifier is used to identify the target data block in the object storage server;

[0007] The target block identifier is returned to the first client so that the first client writes the target data into the target data block; wherein, the target data includes: metadata generated when the first client performs a write operation on the target file;

[0008] The target block identifier is written into the inode of the target file, so that a second client accessing the target file can access the target data in the target data block from the object storage server based on the target block identifier written in the inode.

[0009] This application also provides a data management method for a distributed file system, applied to an object storage server, the method comprising:

[0010] In response to a write request from a first client carrying a target block identifier, target data written by the first client is stored in the target data block corresponding to the target block identifier; wherein, the target block identifier is a block identifier in an idle state allocated by the metadata server to the target file in response to the first client's request to open the target file; the target data includes: metadata generated when the first client performs a write operation on the target file;

[0011] In response to an access request from a second client carrying the target block identifier, the target data in the target data block corresponding to the target block identifier is fed back to the second client; wherein, the second client obtains the target block identifier through the inode of the target file.

[0012] This application also provides a data management device for a distributed file system, applied to a metadata server, the device comprising:

[0013] The block identifier allocation module is used to allocate a target block identifier that is in an idle state to the target file in response to the first client opening the target file in write mode; wherein the target block identifier is used to identify the target data block in the object storage server;

[0014] A block identifier return module is used to return the target block identifier to the first client, so that the first client writes the target data into the target data block; wherein, the target data includes: metadata generated when the first client performs a write operation on the target file;

[0015] A block identifier writing module is used to write the target block identifier into the inode of the target file, so that a second client accessing the target file can access the target data in the target data block from the object storage server based on the target block identifier written in the inode.

[0016] This application also provides a data management device for a distributed file system, applied to an object storage server, the device comprising:

[0017] The data writing module is used to respond to a write request from a first client carrying a target block identifier, and to store the target data written by the first client in the target data block corresponding to the target block identifier; wherein, the target block identifier is a block identifier in an idle state allocated by the metadata server to the target file in response to the first client's request to open the target file; the target data includes: metadata generated when the first client performs a write operation on the target file;

[0018] The data access module is used to respond to an access request from a second client carrying the target block identifier, and to return the target data in the target data block corresponding to the target block identifier to the second client; wherein the second client obtains the target block identifier through the inode of the target file.

[0019] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the data management method of any of the above-described distributed file systems.

[0020] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described distributed file system data management methods.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described distributed file system data management methods.

[0022] This application eliminates the need for the first client to write metadata through a metadata server. Instead, metadata is written to the target data block in the object storage server based on the target block identifier. The metadata server no longer needs to handle high-frequency metadata write requests. When a large number of clients simultaneously write to the same target file, all metadata write requests are distributed across the data blocks of the object storage server, rather than being concentrated on a single metadata server, fundamentally breaking the single-point bottleneck of the metadata server. In this process, by using metadata and file data together as target data and uniformly writing them to the data blocks of the object storage server, strong consistency across data can be achieved. Furthermore, by writing the target block identifier to the inode of the target file, the metadata server allows the second client accessing the target file to directly access the target data in the target data block from the object storage server based on the target block identifier in the inode. This offloads the query pressure for the latest file status from a single metadata server and distributes it across the entire object storage server, increasing the system's concurrent access capability from being limited to a single metadata server to the entire scalable object storage server, thereby significantly improving the overall throughput of the system. Furthermore, this application stores the latest target data of the target file in the target data block, which can be directly and accurately read after a failure without the need for a full scan, thus achieving fast and on-demand data recovery, significantly shortening business interruption time, and avoiding impact on the stability and continuity of upper-layer applications. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A data access diagram provided for an embodiment of this application;

[0025] Figure 2 A flowchart illustrating a data management method for a distributed file system provided in this application embodiment;

[0026] Figure 3 Another data access diagram provided for an embodiment of this application;

[0027] Figure 4 A flowchart illustrating another data management method for a distributed file system provided in this application embodiment. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0029] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] In various fields such as the Internet, high performance, and artificial intelligence, distributed file systems are widely used for storing unstructured data due to their scalability and high performance, which generate a large amount of unstructured data every day. In real-world production environments, distributed file systems, when interfacing with upper-layer applications, not only need to provide data storage and read / write functions, but more importantly, they must possess robust fault recovery capabilities to ensure rapid data access recovery in the event of disk or node failures, thus guaranteeing business continuity and stability.

[0032] Distributed file systems typically employ an architecture that separates metadata and data. The metadata server (MetaDataService, MDS) manages the metadata of all files, while the object storage server (Object Service Device, OSD) manages the file data itself. However, different upper-layer applications generate different I / O models, and the centralized management of distributed file systems exposes significant shortcomings when facing specific I / O models. A typical scenario is when multiple clients concurrently access the same file (e.g., in marine meteorological simulation calculations). During the calculation process, multiple clients will access the same file concurrently to obtain the latest metadata. All queries and update requests for the metadata of these files must be processed by the single metadata server managing that file. This immediately makes the metadata server a single point of performance bottleneck for the entire system, leading to a sharp increase in access latency and severely limiting the overall throughput of the system.

[0033] Furthermore, for performance reasons, clients are typically allowed to cache metadata changes locally. This means metadata isn't sent to the metadata server in real-time, but rather periodically synchronized in batches for persistent storage. However, if a failure occurs during the metadata caching period, this unsent metadata will be lost. To reconstruct this lost metadata, a full scan of the massive object storage server is performed during the metadata server recovery phase to infer and repair metadata file by file. This process is extremely time-consuming, and the larger the number and size of the files, the longer the recovery time. Moreover, the relevant files can only be accessed by upper-layer applications after the metadata is fully recovered. The recovery speed of the metadata server and files directly determines the recovery speed of the upper-layer applications, thus prolonging the system's fault recovery time and failing to meet the urgent needs of critical business operations for rapid fault recovery.

[0034] As a concrete example, in current distributed file systems, file metadata is consistently managed by a single metadata server, and all metadata requests for the same file are sent to that dedicated metadata server. For example... Figure 1As shown, when only one client (e.g., client A) accesses a file, the generated metadata and data can be cached locally on client A, achieving efficient access. When a second client (e.g., client B) also accesses the same file, to ensure data consistency, client A needs to flush all its locally cached metadata to the metadata server for persistence. Only then can client B read the latest metadata from the metadata server and cache it locally. This mechanism means that in scenarios with concurrent access from multiple clients, long-term caching of metadata and data on the client's local machine is not possible. Every metadata update operation by any client triggers all subsequent accesses from all other clients to interact with the metadata server. This undoubtedly puts significant pressure on the metadata server, making it a performance bottleneck and leading to increased overall access latency and decreased throughput.

[0035] More seriously, because metadata needs to be cached locally before being persisted to the metadata server, a failure during the caching process can lead to the loss of metadata that has not yet been persisted. The lost metadata must be recovered before upper-layer applications can access it, and the speed of metadata recovery directly determines the speed of application recovery. However, using a full scan of the object storage server to recover metadata is very slow, prolonging the service interruption time of upper-layer applications due to data unavailability.

[0036] Therefore, under the above circumstances, distributed file systems will suffer from poor access performance and slow business recovery after failure, affecting the stability and continuity of upper-layer applications.

[0037] To address the above issues, this application provides a data management method, device, medium, and program product for a distributed file system. The data management method for the distributed file system will be described in detail below.

[0038] This embodiment provides a data management method for a distributed file system, which is applied to a metadata server. (Refer to...) Figure 2 The data management methods of a distributed file system may include the following steps.

[0039] S102, in response to the first client opening the target file in write mode, allocate a target block identifier in an idle state to the target file; wherein, the target block identifier is used to identify the target data block in the object storage server.

[0040] In this embodiment, the distributed file system mainly includes a metadata server and an object storage server. The object storage server's preset storage space is divided into multiple data blocks of a preset size, and each data block includes a block identifier. The object storage server manages the mapping relationship between the data blocks and the block identifiers.

[0041] Combination Figure 3 In an object storage server, a contiguous storage space is reserved, and this space is divided into a series of logically independent data blocks of a fixed size. For example, an 8KB storage space in the object storage server can be divided into a series of independent data blocks. These blocks will be dynamically allocated to files by the metadata server when a file is accessed, serving as a unified storage area for reading and writing metadata and file data.

[0042] Each data block is assigned a globally unique block identifier, such as an incrementing sequence number, which serves as the logical address for accessing that data block. The object storage server maintains and manages the mapping relationship between data blocks and block identifiers. This mapping relationship can be recorded in the form of tables or similar formats, showing the mapping from each block identifier to its corresponding data block, thereby enabling access to data blocks via block identifiers.

[0043] The databases and their block identifiers in the object storage server are visible globally, and all metadata servers within the cluster share these data blocks and block identifiers. The object storage server can allocate the block identifiers it manages to multiple metadata servers according to preset partitioning rules, allowing each metadata server to determine its own set of original block identifiers. The partitioning rules indicate that the server IDs of different metadata servers correspond to different sets of block identifiers within the object storage server.

[0044] Specifically, all block identifiers managed by the object storage server are distributed among the various metadata servers in the cluster according to partitioning rules. These partitioning rules ensure that the server IDs of different metadata servers correspond to different ranges of block identifiers, thereby guaranteeing conflict-free and load-distributed block identifier management globally.

[0045] For any given metadata server, it uses its own server ID as a logical partition to manage a contiguous and unique set of block identifiers assigned to it globally, which is also the original block identifier set belonging to that cloud data server. The partitioning rules ensure that each metadata server manages a non-overlapping subset of block identifiers, namely its own original block identifier set.

[0046] In a metadata server, all the original block identifier sets assigned to it can be managed and maintained using a KV (Key-Value Pair) method; where K is the server ID of the metadata server and V is the original block identifier set.

[0047] When the metadata server starts, it reads the full set of raw block identifiers from the object storage server and loads it into memory. It then marks the current allocation status of each block identifier in the raw block identifier set, such as allocated or idle. The metadata server manages the correspondence between each block identifier in the raw block identifier set and its current allocation status using a key-value (KV) approach; where K is the block identifier and V is the current allocation status of that block identifier, such as allocated or idle.

[0048] According to the above embodiments, the metadata server completes the initialization of the original block identifier set, preparing for subsequent file access and data block allocation.

[0049] When the first client opens the target file in write mode, it sends an open request to the metadata server to which the target file belongs. The metadata server searches for a free target block identifier in its set of raw block identifiers recorded in memory and assigns the target block identifier to the target file. This target block identifier can uniquely locate the target data block in the object storage server.

[0050] After assigning a target block identifier to a target file, the metadata server updates the current allocation status of the target block identifier to "assigned". By updating the current allocation status of the target block identifier, it is clear which block identifiers have been allocated and which block identifiers are idle. This avoids anomalies such as data overwriting and confusion caused by misjudgment of status, and ensures that reserved data blocks can be reused to the maximum extent, thereby supporting a higher number of concurrent file accesses.

[0051] S104, return the target block identifier to the first client so that the first client can write the target data into the target data block; wherein, the target data may include: metadata generated when the first client performs a write operation on the target file; and, the target data may also include: file data generated when the first client performs a write operation on the target file.

[0052] In this embodiment, the metadata server returns the target block identifier to the first client.

[0053] When the first client performs a write operation on the target file, real-time metadata, such as the updated file size and modification time, is generated. This metadata, along with the newly written file data, is encapsulated into the target data. The first client then sends a write request carrying the target block identifier to the object storage server.

[0054] After receiving a write request from the first client, the object storage server can accurately locate the corresponding target data block by parsing the target block identifier. Then, it persistently stores the target data (including metadata and file data) sent by the first client in the target data block.

[0055] In traditional solutions, metadata is cached locally on the client and then periodically persisted to a metadata server. In contrast, the solution provided in this embodiment, through target block identification, enables the writing of target data, including metadata, from the first client into the target data block of the object storage server. On one hand, this offloads the metadata writing load from the metadata server to the object storage server, distributing the metadata writing pressure across the entire object storage server, effectively improving the concurrent write capability of the distributed file system and significantly reducing the latency of metadata operations. Furthermore, any metadata update is immediately written to the shared target data block of the object storage server. In this case, when other clients access the target file, they can directly read from this target data block, always obtaining the latest and consistent metadata, thus ensuring strong consistency.

[0056] On the other hand, the target data block can simultaneously store metadata and file data. In this embodiment, the two I / O operations for metadata and file data in the relevant scheme are merged into a single write operation, thereby greatly reducing the number of disk I / O operations, thus improving the overall throughput of the system and reducing the latency of the first client's write operation. The first client only needs to send a write request to the object storage server to simultaneously complete the persistence of file data and metadata. This directly reduces the number of I / O operations and network interactions by nearly half, and the performance improvement is particularly significant for high-concurrency, small file, or frequently updated metadata scenarios. At the same time, this embodiment synchronously persists metadata and file data in real time, ensuring that metadata and file data succeed or fail simultaneously. As long as this write is successful, it means that both metadata and file data have been persisted and are matched with each other, thus achieving atomic updates and strong consistency, fundamentally avoiding the risk of metadata loss due to client caching.

[0057] S106, Write the target block identifier into the inode of the target file so that the second client accessing the target file can access the target data in the target data block from the object storage server based on the target block identifier written in the inode.

[0058] In this embodiment, after assigning a target block identifier to the target file, the metadata server persistently records the target block identifier in the target file's inode. This allows any subsequent client accessing the target file (such as a second client) to read or write the target file by retrieving the target block identifier from the inode and directly initiating an access request to the object storage server using the target block identifier, thus accurately accessing the target data block containing the latest metadata and file data.

[0059] In this embodiment, when the second client accesses the target file, if it sees from its inode that the target file has been allocated a target block identifier and a target data block, it means that the latest target data (including metadata and file data) of the target file has been uniformly maintained in the target data block of the object storage server. Therefore, the second client does not need to query the metadata server again, and can directly send an access request carrying the target block identifier to the object storage server to read the latest complete target data from the target data block of the object storage server at once. In this way, the second client can directly read the target data from the object storage server by means of the target block identifier, bypassing the metadata server. When there are a large number of concurrent accesses from multiple clients, the metadata requests carried by the metadata server are offloaded to different object storage servers, thereby eliminating single point bottlenecks and greatly improving concurrent access performance and throughput. Furthermore, all second clients accessing the target file obtain data from the same data source (target data block), which naturally ensures the consistency of data read by all second clients.

[0060] To better understand the solution, the data management methods of the distributed file system will be described in more detail below.

[0061] In this embodiment, the following is included: when the metadata server starts up, it determines the set of original block identifiers it manages from all block identifiers managed by the object storage server according to a preset partitioning rule; wherein, the partitioning rule is used to indicate that the server IDs of different metadata servers correspond to different block identifiers in the object storage server.

[0062] The original block identifier set is loaded into memory, and the original management information of each block identifier in the original block identifier set is updated based on the preset allocation management table; wherein, the original management information includes at least: the status of the block identifier is allocated or idle.

[0063] Specifically, upon initial startup or restart after a failure, the metadata server can determine its own set of original block identifiers from all block identifiers managed by the object storage server, based on preset partitioning rules and its own server ID. The metadata server loads the original block identifier set into memory and updates the original management information of each block identifier in the original block identifier set based on a preset allocation management table.

[0064] The allocation management table in this embodiment can be a table that records block identifiers and their original management information in key-value (KV) format. Here, K represents the block identifier, and V represents the original management information. The original management information may include: the current allocation status of the block identifier (e.g., allocated or idle); if allocated, the inode of the target file to which it was allocated; and may also include timestamps and other information.

[0065] When the metadata server starts up for the first time, it is easy to understand that the current allocation status of each block identifier in its original block identifier set should be idle.

[0066] When the metadata server is not starting up for the first time, i.e., restarting after a failure, it needs to know which data block identifiers had been assigned to files before the failure and which data block identifiers were still idle. Based on this, the metadata server can maintain a dedicated, highly available allocation management table. This allocation management table can be stored in the metadata server's own reliable storage or in a shared configuration center accessible to all metadata servers.

[0067] Then, the metadata server reads the persistent, reliable allocation management table from its memory and uses the allocation management table to redetermine the current allocation status of each block identifier in its original block identifier set.

[0068] In one embodiment, step S102 described above may include:

[0069] In response to the first client opening the target file in write mode, search for candidate block identifiers that are in an idle state in the set of original block identifiers recorded in memory; if the number of candidate block identifiers is not less than the number of the first client, determine the target block identifier to be allocated to the target file from the candidate block identifiers.

[0070] In this embodiment, if the metadata server has a sufficient number of idle block identifiers, that is, if the object storage server has a sufficient number of data blocks available for use, then the metadata server uses the original block identifier set recorded in memory to select all idle block identifiers as candidate block identifiers, and then determines the target block identifier to be allocated to the target file from the candidate block identifiers.

[0071] Correspondingly, if the number of idle block identifiers in the metadata server is insufficient—that is, if the number of concurrently accessed files exceeds the number of idle block identifiers—data blocks cannot be allocated to these files. In this case, this embodiment may further include:

[0072] If the number of candidate block identifiers is less than the number of files in the target file, a local caching instruction is sent to the first client, causing the first client to write metadata to its local cache and send the metadata in the local cache to the metadata server for storage according to a preset period. Additionally, the first client sends the file data to the object storage server for storage.

[0073] According to the above embodiments, after the metadata server allocates a target block identifier that is in an idle state to the target file, this embodiment includes: updating the target management information of the target block identifier in a preset allocation management table; wherein, the target management information includes at least: the status of the target block identifier is an allocated state.

[0074] In this embodiment, the data management method of the distributed file system further includes:

[0075] In response to a third client's request to close the target file, determine whether other clients are accessing the target file; if no other clients are accessing the target file, release the target block identifier and target data block, and delete the target block identifier from the inode of the target file.

[0076] It's easy to understand that each client will close the target file after accessing it. Let's assume the client currently closing the target file is the third client. When the third client closes the target file, it sends a close request to the metadata server.

[0077] The metadata server determines whether other clients are accessing the target file, or in other words, whether the third client is the last client.

[0078] As a concrete example, a metadata server can maintain a file open reference counter for open target files, initially set to 0. The counter increments by 1 whenever a client successfully accesses the target file in read / write mode; it decrements by 1 whenever a client initiates a file close request. Based on this, when the counter is 0, it is determined that the third client currently closing the target file is the last client, and no other clients are accessing the target file.

[0079] In this case, the target block identifier and target data block are released, and the target block identifier within the inode of the target file is deleted.

[0080] For example, releasing a target block identifier includes: the metadata server updating the allocation management table, changing the current allocation status of the target block identifier from allocated to idle, and deleting its association record with the inode of the target file, thereby making the target block identifier available for allocation to other new files.

[0081] Delete the target block identifier within the inode of the target file, including: the metadata server locates the inode of the target file, deletes the target block identifier stored therein, and severs the association between the target file and the original target data block.

[0082] The process of notifying the object storage server to release the target data block includes: the metadata server sending a data block cleanup command carrying the target block identifier to the object storage server. After receiving the command, the object storage server clears the target data in the target data block corresponding to the target block identifier, marks the target data block as idle, and waits for it to be associated with a new block identifier for subsequent use.

[0083] In this embodiment, after the last client (i.e., the third client) closes the target file, the target block identifier and the target data block are released, and the target block identifier within the target file's inode is deleted, enabling the cyclic reuse of block identifiers and data blocks. Furthermore, this embodiment releases resources only when the last third client closes the target file, avoiding the problem of clients accidentally releasing resources when closing the target file, which could cause data access failures for other clients currently accessing the target file, thus ensuring the continuity of concurrent access from multiple clients.

[0084] In this embodiment, the data management method of the distributed file system further includes:

[0085] After a failure occurs in the distributed file system, in response to the first access to the target file by the fourth client, the latest data is read from the target data block corresponding to the target block identifier by the fourth client, and the latest data is used to update and recover.

[0086] During the operation of a distributed file system, hardware failures and software anomalies can easily lead to system failures. Failure recovery can be divided into two stages: metadata server recovery and file recovery.

[0087] Metadata server recovery includes: when the metadata server recovers and starts up after a failure, it determines the set of original block identifiers it manages from all block identifiers managed by the object storage server according to the preset partitioning rules; loads the set of original block identifiers into memory, and updates the original management information of each block identifier in the set of original block identifiers based on the preset allocation management table; wherein the original management information includes at least: the status of the block identifier is allocated or idle.

[0088] The metadata server recovery phase described above does not require scanning and confirming files with lost metadata, thus this process is omitted. Regarding the file recovery phase, this embodiment does not require metadata recovery during the cluster recovery process; this step is skipped. Instead, recovery can be triggered when the upper-layer application first accesses the target file.

[0089] During the recovery process, after the fourth client restarts the metadata server, it first attempts to access a target file that was opened and written to before the failure, but whose metadata was not fully synchronized to the metadata server. The fourth client retrieves the unique target block identifier allocated before the failure from the target file's inode. Instead of directly requesting potentially outdated metadata from the metadata server, the fourth client sends a recovery request directly to the object storage server, reading the latest metadata and file data saved by the last writer before the failure from the target data block corresponding to the target block identifier.

[0090] The fourth client sends the latest metadata it reads to the metadata server. The metadata server uses this metadata to update the main metadata of the target file in memory, thus completing the metadata repair of the target file.

[0091] The data recovery solution provided in this embodiment does not require scanning fragment information on the object storage server. Instead, it directly reads the latest metadata and file data from the target data block on the object storage server according to the target block identifier allocated to the target file, directly locating and reading the lost data, saving a lot of scanning time and improving recovery speed.

[0092] In summary, the data management method for a distributed file system provided in the above embodiments, applied to a metadata server, includes: in response to a first client opening a target file in write mode, allocating a target block identifier in an idle state to the target file; wherein the target block identifier is used to identify a target data block in an object storage server; returning the target block identifier to the first client so that the first client writes target data into the target data block; wherein the target data includes: metadata generated when the first client performs a write operation on the target file; and writing the target block identifier into the inode of the target file so that a second client accessing the target file can access the target data in the target data block from the object storage server based on the target block identifier written in the inode.

[0093] In this technical solution, the metadata server establishes a write and access channel between the client and the object storage server by assigning target block identifiers to the target file. This offloads the high-frequency operations of writing and accessing target data, including metadata, from the metadata server to the object storage server. In scenarios where multiple clients concurrently access the same target file, the metadata load is significantly reduced by offloading it from the metadata server to the object storage server. The metadata server no longer needs to handle a large number of concurrent metadata requests, greatly improving concurrency and access performance. Simultaneously, the latest target data of the target file is stored in the target data block, allowing for direct and accurate reading after a failure without a full scan. This enables rapid, on-demand data recovery, significantly shortening business interruption time and preventing impact on the stability and continuity of upper-layer applications.

[0094] Specifically, the metadata from the first client is no longer written through the metadata server. Instead, it is written to the target data block in the object storage server based on the target block identifier. The metadata server no longer needs to handle high-frequency metadata write requests. When a large number of clients write to the same target file simultaneously, all metadata write requests are distributed across the data blocks of the object storage server, rather than being centralized on a single metadata server, fundamentally breaking the single point of failure bottleneck of the metadata server. In this process, by using both metadata and file data as target data and uniformly writing them to the data blocks of the object storage server, strong consistency across data can be achieved.

[0095] Furthermore, by writing the target block identifier into the inode of the target file, the metadata server enables a second client accessing the target file to directly access the target data in the target data block from the object storage server based on the target block identifier in the inode. In this way, the query pressure for the latest file status is offloaded from a single metadata server and distributed to the entire object storage server, increasing the system's concurrent access capability from being limited to a single metadata server to the entire scalable object storage server, thereby greatly improving the overall throughput of the system.

[0096] Corresponding to the aforementioned data management method for distributed file systems applied to metadata servers, refer to Figure 4 This embodiment provides a flowchart of a data management method for a distributed file system. This method is applied to an object storage server and includes:

[0097] S202, in response to the write request from the first client carrying the target block identifier, the target data written by the first client is stored in the target data block corresponding to the target block identifier; wherein, the target block identifier is a block identifier that is in an idle state allocated by the metadata server to the target file in response to the first client's request to open the target file; the target data includes: metadata generated when the first client performs a write operation on the target file;

[0098] S204, in response to the access request from the second client carrying the target block identifier, the target data in the target data block corresponding to the target block identifier is fed back to the second client; wherein, the second client obtains the target block identifier through the inode of the target file.

[0099] In one embodiment, the method further includes:

[0100] In the preset storage space of the object storage service server, the data is divided into multiple data blocks according to a preset size. Each data block includes a block identifier. The mapping relationship between the data blocks and the block identifiers is managed.

[0101] According to the preset partitioning rules, the block identifiers managed by the object storage server are assigned to multiple metadata servers, so that each metadata server can determine the original block identifier set it manages; wherein, the partitioning rules are used to indicate that the server IDs of different metadata servers correspond to different block identifier sets in the object storage server.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0103] Embodiments of this application also provide a data management device for a distributed file system, which is applied to a metadata server, and the device includes:

[0104] The block identifier allocation module is used to allocate a target block identifier that is in an idle state to the target file in response to the first client opening the target file in write mode; wherein the target block identifier is used to identify the target data block in the object storage server;

[0105] A block identifier return module is used to return the target block identifier to the first client, so that the first client writes the target data into the target data block; wherein, the target data includes: metadata generated when the first client performs a write operation on the target file;

[0106] A block identifier writing module is used to write the target block identifier into the inode of the target file, so that a second client accessing the target file can access the target data in the target data block from the object storage server based on the target block identifier written in the inode.

[0107] Embodiments of this application also provide a data management apparatus for a distributed file system, which is applied to an object storage server. The apparatus includes:

[0108] The data writing module is used to respond to a write request from a first client carrying a target block identifier, and to store the target data written by the first client in the target data block corresponding to the target block identifier; wherein, the target block identifier is a block identifier in an idle state allocated by the metadata server to the target file in response to the first client's request to open the target file; the target data includes: metadata generated when the first client performs a write operation on the target file;

[0109] The data access module is used to respond to an access request from a second client carrying the target block identifier, and to return the target data in the target data block corresponding to the target block identifier to the second client; wherein the second client obtains the target block identifier through the inode of the target file.

[0110] For a description of the features in the embodiments corresponding to the data management device of the distributed file system, please refer to the relevant descriptions in the embodiments corresponding to the data management method of the distributed file system, which will not be repeated here.

[0111] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the data management method for a distributed file system.

[0112] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the data management method for a distributed file system.

[0113] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0114] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the data management method for a distributed file system.

[0115] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described distributed file system data management method embodiments.

[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The data management method and device for a distributed file system provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data management method of a distributed file system, characterized by, The method applied to a metadata server comprises: In response to a first client opening a target file in a write mode, a target block identifier in an idle state is allocated to the target file; wherein the target block identifier is used to identify a target data block in an object storage server; the object storage server is divided into a plurality of data blocks, each of which is allocated a unique block identifier, and the data blocks are used to be allocated to a file by the metadata server as a unified storage area of read-write metadata and file data when the file is accessed; The target block identifier is returned to the first client to enable the first client to write target data into the target data block; wherein the target data includes metadata generated when the first client performs a write operation on the target file; The target block identifier is written into an inode of the target file to enable a second client accessing the target file to access the target data block in the target data according to the target block identifier written in the inode.

2. The data management method of a distributed file system according to claim 1, wherein, The method of allocating the target block identifier in the idle state to the target file in response to the first client opening the target file in the write mode comprises: In response to the first client opening the target file in the write mode, a candidate block identifier in an idle state is searched for in a set of original block identifiers recorded in memory; In a case where the number of the candidate block identifiers is not less than the number of the first client, a target block identifier allocated to the target file is determined from the candidate block identifiers.

3. The data management method of a distributed file system according to claim 1, wherein, After the target block identifier in the idle state is allocated to the target file, the method further comprises: Target management information of the target block identifier is updated in a preset allocation management table; wherein the target management information at least includes a state of the target block identifier being an allocated state.

4. The data management method of a distributed file system according to claim 1, wherein, The method further comprises: When starting, the metadata server determines a set of original block identifiers managed by itself from all block identifiers managed by the object storage server according to a preset partition rule; wherein the partition rule is used to indicate that different server IDs of the metadata server correspond to different block identifiers in the object storage server; The set of original block identifiers is loaded into memory, and original management information of each block identifier in the set of original block identifiers is updated based on a preset allocation management table; wherein the original management information at least includes a state of the block identifier being an allocated state or an idle state.

5. The data management method of a distributed file system according to claim 1, wherein, The method further comprises: In response to a close request of the target file by a third client, it is judged whether there is another client accessing the target file; In a case where there is no other client accessing the target file, the target block identifier and the target data block are released, and the target block identifier in the inode of the target file is deleted.

6. The data management method of a distributed file system according to claim 1, wherein, The method further comprises: After a distributed file system fails, in response to a fourth client first accessing the target file, the latest data is read from the target data block corresponding to the target block identifier by the fourth client, and the latest data is used for update recovery.

7. The data management method of a distributed file system according to any one of claims 1 to 6, characterized in that, The target data further includes file data generated when the first client performs a write operation on the target file.

8. A data management method of a distributed file system, characterized by, The method is applied to an object storage server, and the method comprises: In response to a write request carried by a first client and carrying a target block identifier, storing target data written by the first client in a target data block corresponding to the target block identifier; wherein the target block identifier is a block identifier in an idle state allocated by a metadata server to a target file in response to an open request of the first client on the target file; the target data includes metadata generated when the first client performs a write operation on the target file; the object storage server is divided into a plurality of data blocks, each of the data blocks is allocated a unique block identifier, and the data blocks are used to be allocated to a file by the metadata server as a unified storage area of read-write metadata and file data when the file is accessed; In response to an access request carried by a second client and carrying the target block identifier, feeding back the target data in the target data block corresponding to the target block identifier to the second client; wherein the second client obtains the target block identifier through an inode INODE of the target file.

9. The data management method of a distributed file system according to claim 8, wherein, The method further comprises: In a preset storage space of the object storage server, a plurality of data blocks are divided according to a preset size, and the data blocks include block identifiers; A mapping relationship between the data blocks and the block identifiers is managed; According to a preset partition rule, the block identifiers managed by the object storage server are allocated to a plurality of metadata servers, so that each of the metadata servers determines a set of original block identifiers managed by itself; wherein the partition rule is used to represent that a server ID of a different metadata server corresponds to a different set of block identifiers in the object storage server.

10. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the data management method of the distributed file system according to any one of claims 1 to 9.

Images