Data processing method and apparatus in distributed file system, and storage cluster

Abstract

The present application belongs to the technical field of storage. Provided are a data processing method and apparatus in a distributed file system, and a storage cluster. In the method, on the basis that a distributed file system stores a log of a file in a cache space, persistent storage is performed on the log on the basis of whether the data volume of the log in the cache space meets a condition. During this process, when the data volume of a certain log does not meet the condition, the log is redirected to release an initial position of the log in the cache space, such that the distributed file system can continue to process the next log of the log. Therefore, the possibility of aggregating discrete data over a relatively long time span is provided. The method is applicable to any type of file. Moreover, since the persistent storage process is performed on the basis of the data volume of the log, the log which meets the condition is persistently stored, thereby avoiding the fragmentation of a persistent space, and improving the access performance of the system.

Description

Data processing methods, devices, and storage clusters in distributed file systems

[0001] This application claims priority to Chinese Patent Application No. 202411689490.0, filed on November 22, 2024, entitled "Data Processing Method, Apparatus and Storage Cluster in Distributed File System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of storage technology, and in particular to a data processing method, apparatus and storage cluster in a distributed file system. Background Technology

[0003] When using write-ahead log (WAL) technology in a distributed file system, the system first records file write operations as logs and stores these logs in the line space (LS). At a later point, the logs in the LS are persisted one by one according to the cursor and sliding window, and the LS storage space is gradually released, improving the system's data write performance. In some scenarios, such as when multiple file write operations are concurrent, the logs in the LS are often random and unordered. Since the persistence process of the WAL mechanism relies on the LS cursor and sliding window to proceed step by step, the system will frequently persist random and unordered logs, causing fragmentation of the persistence space and impacting system access performance.

[0004] In related technologies, an optimization scheme is proposed to address the problems of the WAL mechanism mentioned above: For logs that are not persistently stored in the system, the system maintains the association between these logs. When the cursor of LS points to a log (such as log1), according to the association between logs, other logs that can form a continuous log with log1 (such as log2) are found. log1 and log2 are persistently stored together and the log status of log2 in LS is marked. When the cursor continues to move to log2, the persistent storage of log2 can be skipped according to the log status of log2, and the storage space occupied by log2 in LS is released.

[0005] However, the above solution only applies to files that can form continuous logs in a short period of time. For files that cannot form continuous logs in a short period of time, it will still cause fragmentation of the persistent space, resulting in poor system access performance. Summary of the Invention

[0006] This application provides a data processing method, apparatus, and storage cluster in a distributed file system, which can avoid fragmentation of persistent space in the distributed file system and improve system access performance.

[0007] Firstly, a data processing method in a distributed file system is provided, applicable to scenarios where the distributed file system employs a Write-Ahead Logging (WAL) mechanism to improve data write performance. This method includes:

[0008] Logs of files in the distributed file system are stored in a cache space. The logs are used to record operations on the files. Based on the amount of log data in the cache space, the logs in the cache space are persisted.

[0009] Specifically, if the amount of data in the first log in the cache space does not meet the first condition, the first log is redirected to release its initial position in the cache space.

[0010] In this context, "file" refers to any file within a distributed file system. This application does not limit the type of data within the file; it may include, for example, text, images, audio, video, etc., but is not limited to these. The operations performed on a file are illustrative, such as reading, writing, deleting, and modifying, but are not limited to these.

[0011] In the above method, based on the distributed file system storing file logs in the cache space, logs are persisted based on whether the amount of log data in the cache space meets the conditions. During this process, when the amount of data in a log does not meet the conditions, the log is redirected to release its initial position in the cache space, allowing the distributed file system to continue processing the next log. This provides the possibility of aggregating discrete data over a long time span, is applicable to any type of file, and since the persistence process is based on the amount of log data, logs that meet the conditions will be persisted, thereby avoiding fragmentation of the persistence space and improving system access performance.

[0012] In one possible implementation, the logs in the cache space are persistently stored based on the amount of data in the cache space. This includes: determining that the amount of data in the second log in the cache space meets a first condition, and then persistently storing the second log. Based on this, logs whose data amount meets the first condition will be persistently stored. In other words, the amount of data in the logs written to the persistent space meets the first condition, thus avoiding fragmentation of the persistent space and making it applicable to any type of file.

[0013] In one possible implementation, the method further includes: determining that the dispersion of logs in the cache space satisfies a second condition, and persistently storing the third log of the file in the cache space, wherein the dispersion is used to indicate the number of logs in the cache space that cannot form continuous logs with other logs.

[0014] This approach provides a discrete-degree-of-dispersion (DCD)-based persistent storage mechanism that avoids discrete data from occupying cache space for extended periods. For example, in certain I / O models, even if all logs undergo repeated redirection until the cache space is exhausted, the first condition may still not be met. In this case, the DCD is used to measure the proportion of logs in the cache space that still cannot meet the first condition despite repeated redirection, thereby determining whether some logs that do not meet the first condition need to be persisted and cache space released in a timely manner.

[0015] In one possible implementation, the third log refers to a log that has already undergone redirection. That is, when there is a lot of discrete data in the cache space, a log that has undergone redirection can be forced to be persisted. Since the third log has already undergone redirection, there is a high probability that there will be no logs in the cache space that form a continuous log with the third log in the short term. By persisting the third log, the cache space can be released in a timely manner.

[0016] In one possible implementation, determining that the dispersion of logs in the cache space satisfies a second condition, and then persistently storing the third logs of the files in the cache space, including:

[0017] If the dispersion of logs in the cache space satisfies the second condition and there are multiple third logs, determine the fourth log that can form a continuous log with multiple third logs;

[0018] The fourth log is read from the persistent storage space. Multiple third logs and the read fourth log are merged into the fifth log. The fifth log is then stored persistently. The fifth log is a continuous log.

[0019] The above process means that when there is a lot of discrete data in the cache space and there are multiple third logs to be persisted, since the third logs are logs that have already been redirected, it indicates that there may be consecutive logs that have already been persisted. Therefore, these logs can be read from the persistent storage space, and the read logs can be merged with the third logs to obtain the fifth log. The fifth log is then persisted. In this way, the discrete logs in the cache space are concatenated into consecutive logs before being persisted, avoiding fragmentation of the persistent space and improving the access performance of the persistent space.

[0020] In one possible implementation, after the fourth log is read from the persistent storage space, the fourth log in the persistent storage space is reclaimed. By reclaiming the fourth log in the persistent space, the persistent space is released in a timely manner, improving disk space utilization.

[0021] In one possible implementation, determining that the amount of data in the first log in the cache space does not meet a first condition, and redirecting the first log to release its initial position in the cache space, includes:

[0022] If the data volume of the first log entry does not meet the first condition, redirect the first log entry to the target location in the cache space and release the initial location. The target location is different from the initial location. Alternatively...

[0023] If the data volume of the first log does not meet the first condition, the first log and the sixth log are merged into the seventh log to redirect the first log to another log other than the first log. The initial position is released and the seventh log is persisted. The data volume of the seventh log meets the first condition and the seventh log is a continuous log.

[0024] By redirecting the first log to the target location in the cache space, when the distributed file system receives an access request for the data corresponding to the first log, it can quickly read the corresponding data from the cache space without accessing the persistent space, greatly improving data access performance. Furthermore, by merging the first and sixth logs into a seventh log that satisfies the first condition and persistently storing the seventh log, the discrete data in the cache space is merged, avoiding fragmentation of the persistent space.

[0025] In one possible implementation, determining that the data volume of the first log does not meet the first condition, redirecting the first log to a target location in the cache space, and releasing the initial location includes:

[0026] The first log entry is stored at the end of the cache space in an append-only manner. This way, when the distributed file system receives an access request for data corresponding to the first log entry, it can quickly read the corresponding data from the cache space without accessing the persistent space, greatly improving data access performance; or...

[0027] The first and eighth logs are merged into the ninth log, which is then appended to the end of the cache space. The ninth log is a continuous log, and its data size does not meet the first condition. By merging the first and eighth logs into the ninth log, the discrete data in the cache space is consolidated, facilitating subsequent persistent storage. Furthermore, the ninth log, which does not meet the first condition, is retained in the cache space for as long as possible. This ensures that when the distributed file system receives an access request for the data corresponding to that log, it can quickly retrieve the data from the cache space without accessing persistent storage, significantly improving data access performance.

[0028] In one possible implementation, the method further includes:

[0029] Query the management information of the cache space. The management information is used to indicate whether different logs of a file can form a continuous log.

[0030] Storing the first log in an append-only manner to the end of the cache space includes: determining that the management information indicates that there is no log in the cache space that can form a continuous log with the first log, and storing the first log in an append-only manner to the end of the cache space.

[0031] Merging the first log and the eighth log into the ninth log includes: determining that there is a log in the management information indication cache space that can form a continuous log with the first log, and merging the first log and the eighth log into the ninth log.

[0032] The above process is as follows: when it is determined that the amount of data in the first log does not meet the first condition, the management information of the cache space is queried. If the management information indicates that there is no log in the cache space that can form a continuous log with the first log, the first log is directly rewritten to the cache space in an append-only manner. If the management information indicates that there is a log in the cache space that can form a continuous log with the first log, the first log and the log that can form a continuous log are merged, and the continuous log obtained after merging is written to the cache space in an append-only manner.

[0033] In one possible implementation, merging the first log and the eighth log into the ninth log involves removing duplicates from the overlapping portions of the first and eighth logs before merging them into the ninth log. By removing duplicates from the overlapping portions during the redirection process, cache space usage can be saved, improving the utilization rate of storage resources.

[0034] In one possible implementation, the first condition refers to the log data volume reaching the striping threshold of the distributed file system. The striping threshold is used to indicate the granularity at which the distributed file system performs striping operations. By setting the first condition to the data volume reaching the striping threshold, logs with data volumes reaching the striping threshold are persistently stored, which can significantly improve the utilization of persistent storage space and enhance data access efficiency.

[0035] In one possible implementation, the cache space is a non-volatile storage medium of the storage nodes in the distributed file system. Non-volatile storage media include, but are not limited to, non-volatile dual in-line memory modules (NVDIMMs), storage class memory (SCM), or solid-state drives (SSDs). It should be understood that such non-volatile storage media typically possess high-speed read / write performance and can serve as a cache to accelerate data read / write operations.

[0036] In one possible implementation, the storage node is the access node or home node of the distributed file system. That is, this application does not limit the type of storage node corresponding to the cache space, and the solution has universality.

[0037] In a second aspect, a data processing apparatus in a distributed file system is provided, the apparatus comprising at least one functional module for performing a data processing method of a distributed file system as provided in the first aspect or any possible implementation thereof.

[0038] Thirdly, a storage cluster is provided, the storage cluster including at least one storage node, each storage node including a processor and a memory, the processor of the at least one storage node being used to execute instructions stored in the memory of the at least one storage node, so that the storage cluster performs the data processing method of the distributed file system provided by the first aspect or any possible implementation of the first aspect.

[0039] Fourthly, a computer program product containing instructions is provided, which, when executed by a storage cluster, causes the storage cluster to perform a data processing method of a distributed file system as provided in the first aspect or any possible implementation thereof.

[0040] Fifthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a storage cluster, enable the storage cluster to perform a data processing method of a distributed file system as provided in the first aspect or any possible implementation thereof.

[0041] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0042] Figure 1 is a schematic diagram of a write-ahead log (WAL) mechanism;

[0043] Figure 2 is a schematic diagram of a distributed file system using erasure coding (EC) mechanism;

[0044] Figure 3 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0045] Figure 4 is a schematic diagram illustrating the principle of a redirection method based on the WAL mechanism provided in an embodiment of this application;

[0046] Figure 5 is a flowchart of a data processing method in a distributed file system provided in an embodiment of this application;

[0047] Figure 6 is a schematic diagram of a discrete-degree-based persistent storage mechanism provided in an embodiment of this application;

[0048] Figure 7 is a flowchart of another data processing method in a distributed file system provided in an embodiment of this application;

[0049] Figure 8 is a schematic diagram of the structure of a data processing device in a distributed file system provided in an embodiment of this application;

[0050] Figure 9 is a schematic diagram of the structure of a storage node provided in an embodiment of this application;

[0051] Figure 10 is a schematic diagram of a storage cluster provided in an embodiment of this application;

[0052] Figure 11 is a schematic diagram of a possible implementation of a storage cluster provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the documents, logs, operations, etc. involved in this application were all obtained under fully authorized conditions.

[0054] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0055] A file system is a way to store and organize data files. Typically, a file consists of data and metadata. Data is the actual content of the file, while metadata contains descriptive information such as filename, size, creation time, modification time, and access permissions. The concept of a "file" organizes data in a computer, grouping data used for the same purpose into different types of files according to the structural requirements of different applications.

[0056] A distributed file system is a file system that distributes data across multiple storage nodes. It improves system access performance through the concurrency capabilities of the multiple nodes and increases the overall system capacity through the capacity of the multiple nodes. Storage nodes can be, for example, physical servers.

[0057] Write-ahead log (WAL) is a data management mechanism that records data changes in the form of a log before the actual modifications are made. Illustratively, when a data write operation occurs, the data is not directly written to persistent storage. Instead, the write operation is first logged to a file called the WAL. If the write fails, the record in the WAL is ignored, and the modification is rolled back. If the write succeeds, the log in the WAL will be written to persistent storage at a later time, and the space occupied by that log entry in the WAL will be released.

[0058] Line space (LS) is a virtual, contiguous data buffer space with logical addresses starting from 0 and extending to infinity. LS provides a northbound interface (e.g., an append-only write interface, allowing only append write operations), allowing data to be appended to the LS regardless of the details of the underlying persistent media. LS maintains the amount of data that can reside in the LS simultaneously through a sliding window, and establishes a mapping between the logical addresses of the LS and the physical addresses of the underlying persistent media. Illustratively, the Write-Ahead Log (WAL) mechanism typically uses the LS to store logs. For example, referring to Figure 1, Figure 1 is a schematic diagram of a write-ahead log (WAL) mechanism. As shown in Figure 1, the distributed file system records write operations to files as logs, stores the logs in the linear space (LS), and then, based on the cursor and sliding window of the LS (i.e., the logical space available in the LS), persists the logs in the LS one by one (or flushes them to the persistent space), gradually releasing the storage space of the LS to improve the system's data write performance.

[0059] Erasure coding (EC) is a data redundancy protection mechanism widely used in distributed storage. EC encodes the original data to generate a certain amount of redundant data, allowing the original data to be recovered from the remaining data and the redundant data even if some data is lost. For example, in a distributed file system, N data blocks and M parity blocks (or redundant data blocks) are used to ensure data reliability (N and M are both positive integers). If any M data blocks out of N+M data blocks are corrupted, the remaining M data blocks can be recovered from the other N data blocks. Compared to replication storage, EC provides high reliability while also offering higher disk utilization and reducing costs.

[0060] Striping and striping are concepts in the field of distributed storage. A strip refers to the smallest unit of data that is divided into; striping refers to the process of dividing data into multiple stripes and distributing them across different storage nodes. By setting a striping threshold, the system can determine when to perform striping operations; in other words, the striping threshold indicates the granularity at which the system performs striping operations. For example, with a striping threshold of 2MB and a stripe size of 512KB, when the file size reaches 2MB, the system will divide the file into four 512KB stripes and distribute these four stripes across different storage nodes.

[0061] To illustrate, when using the EC mechanism in a distributed file system, taking an EC ratio of 4+2 as an example (that is, using 4 data blocks and 2 parity blocks to ensure data reliability), the meaning of stripes and stripes can be seen in Figure 2. Figure 2 is a schematic diagram of a distributed file system using the EC mechanism. As shown in Figure 2, taking a stripe threshold of 2MB as an example, the file is divided into 4 data blocks, each data block is called a stripe, and the data size of each stripe is 512KB (or the stripe size is 512KB). In addition, 2 parity blocks are generated by encoding the 4 data blocks. In the figure, strip1 to strip6 together constitute a persistent data unit, that is, the actual amount of data written to the disk is 512KB × 4 (valid data) + 512KB × 2 (redundant data) = 3MB. It should be understood that in a distributed file system using the EC mechanism, data read operations or write operations are performed with the stripe threshold as the smallest granularity to ensure data reliability. Therefore, to improve data writing efficiency and access performance, data is usually written to disk in whole stripes. If the written data is fragmented, it will negatively impact system efficiency. For example, when a write operation instructs the writing of 512KB of data, since distributed file systems using the EC mechanism also require writing in whole stripes, only strip1 is valid data. Strip2, strip3, and strip4 need to be padded with zeros. After calculating strip5 and strip6, the entire stripe is written to disk. Thus, only 512KB of data is valid, but 3MB of data is actually written, resulting in severe write amplification. Moreover, if logically consecutive 512KB of file content is then written, another stripe needs to be written to disk following the same logic. When the system reads the total 1MB of consecutive data (512KB + 512KB) written above, it needs to read twice to read both stripes to piece together the complete 1MB of data, leading to low data access efficiency.

[0062] Garbage collection (GC) is an automatic memory management mechanism. Typically, in append-only storage systems, the logical and physical addresses of data are mapped. If data is modified at a logical address, the modified data is written to the physical storage, and the mapping is updated. The older data is marked as "garbage," and this marked data is reclaimed, freeing up the memory space it occupies. Related technologies for GC include, but are not limited to, reference counting, mark-and-sweep, copying, and generational collection algorithms.

[0063] The application scenarios and implementation environment of this application are described below.

[0064] This application is applied to scenarios where a distributed file system uses a WAL mechanism to improve data write performance. However, this application does not limit the type of distributed file system or the type of data stored in the distributed file system.

[0065] In related technologies, when write operations on multiple files are performed concurrently in a distributed file system, the logs in the File System (LS) typically exhibit randomness and unordered behavior. To avoid fragmentation of the persistent storage space, an optimization scheme for the Write-Ahead Log (WAL) mechanism has been proposed: For logs not persistently stored in the system, the system maintains the relationships between these logs. When the LS cursor points to a log (e.g., log1), based on the relationships between logs, it searches for other logs (e.g., log2) that can form a continuous log with log1, and persists log1 and log2 together. Furthermore, the log status of log2 is marked so that when the cursor points to log2, it can directly skip and release the LS storage space. However, this scheme only applies to files that can form a continuous log within a short period of time. For files that cannot form a continuous log within a short period of time, it will still cause fragmentation of the persistent storage space, resulting in poor system access performance.

[0066] Based on this, this application relates to a data processing method in a distributed file system, providing a redirection method based on the WAL mechanism. This method can avoid fragmentation of the persistent space for any type of file and improve system access performance. The implementation environment of this application is described below with reference to Figure 3.

[0067] Figure 3 is a schematic diagram of an implementation environment provided in an embodiment of this application. As shown in Figure 3, the implementation environment includes a client 100 and a distributed file system 200. The client 100 is directly or indirectly connected to the distributed file system 200 through a wireless network or a wired network.

[0068] Client 100 provides users with data storage and data access functions. Indicatively, a user uploads local data to the distributed file system 200 via client 100. The distributed file system 200 then stores the received data in a distributed manner as files, thus achieving data storage functionality. Additionally, users can access files stored on the distributed file system 200 by sending various operation requests to the distributed file system 200 via client 100. These operation requests instruct the user to perform various operations on the files, such as reading, writing, deleting, and modifying, but are not limited to these. The distributed file system 200 receives the operation requests, performs the corresponding operations on the files based on the requests, and returns the results to client 100, thus achieving data access functionality. Client 100 can be deployed on the user's devices, such as application servers, desktop computers, laptops, mobile phones, etc., and this application does not limit its deployment.

[0069] The distributed file system 200 is a file system that distributes data across multiple storage nodes. In this embodiment, the distributed file system 200 uses a Write-Ahead Log (WAL) mechanism to manage files. That is, before actually modifying a file, the modification operation is recorded in the form of a log, the log is stored in a cache space, and then the log in the cache space is persistently stored. Illustratively, the distributed file system 200 includes multiple storage nodes 201, which are interconnected. Any storage node 201 can be, for example, a server, a desktop computer, etc., and this application does not limit its scope. Furthermore, the number of storage nodes 201 shown in the figure is merely illustrative; the number of storage nodes 201 can be more or less, and this application does not limit its scope.

[0070] In some embodiments, the WAL mechanism is implemented by any one of the multiple storage nodes 201 (represented by storage node A in the figure). Indicatively, this storage node 201 is configured with a cache space to store file logs. The logs record operations performed on the files, such as reading, writing, modifying, and deleting. In some embodiments, the cache space is a linear space (LS) implemented using non-volatile storage media. Non-volatile storage media can be, for example, non-volatile dual in-line memory modules (NVDIMMs), storage class memory (SCM), or solid-state drives (SSDs), and is not limited to these. It should be understood that such non-volatile storage media typically possess high-speed read / write performance and can act as a high-speed cache to accelerate data read / write operations.

[0071] In some embodiments, the storage node 201 used to implement the WAL mechanism is an access node or a home node of the distributed file system 200. The access node of the distributed file system 200 is the entry point through which the client 100 connects to the distributed file system 200. The access node is responsible for receiving requests from the client 100 and forwarding these requests to the corresponding storage node for processing, or reading data from the cache space without accessing other storage nodes. For example, the client 100 connects to the distributed file system 200 via a network address, and the server corresponding to that network address is the access node. When the client 100 requests to read a file, the access node forwards the request to the storage node storing that file based on the file's location information. The home node of the distributed file system 200 is the storage node in the distributed file system 200 responsible for managing a specific file; that is, for a given file, there is a clearly defined home node. The home node is also typically responsible for maintaining metadata related to the file, such as the data size, creation time, modification time, and access permissions. For example, in the distributed file system 200, file 1 corresponds to a home node. When the distributed file system 200 receives an operation request for file 1, it will first send the operation request to the home node of file 1. The home node applies the operation request to the copy of the local storage and coordinates the replication of the data to other storage nodes to ensure data reliability.

[0072] In some embodiments, the distributed file system 200 is deployed on a cloud platform, meaning that the data processing method in the distributed file system provided in this application can be executed by the cloud platform. Here, cloud platform is short for cloud computing platform, referring to services based on hardware and software resources that provide computing, networking, and storage capabilities. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to the user, featuring large scale, distributed nature, virtualization, high availability, scalability, on-demand service, and security. Cloud platforms can achieve rapid deployment and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers. Illustratively, a cloud platform is a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In some embodiments, the cloud platform can also be a virtual machine instance, container instance, etc., but this application is not limited to these.

[0073] In some embodiments, the wireless or wired networks described above use standard communication technologies and / or protocols. Networks include, but are not limited to, data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some implementations, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or part of the link. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0074] The data processing method in the distributed file system 200 provided in this application is described below.

[0075] As described above, this application provides a redirection method based on the WAL mechanism, which can avoid fragmentation of persistent space and improve system access performance for files of any type. For ease of understanding, the principle of the WAL-based redirection method will be introduced below with reference to Figure 4. Figure 4 is a schematic diagram illustrating the principle of a WAL-based redirection method provided in an embodiment of this application. As shown in Figure 4, an example of any file in a distributed file system will be used for explanation.

[0076] To illustrate, a distributed file system responds to a client's request to write to a file by performing a write operation on the file. For example, the file write data I / O stream is shown below (offset is the offset, representing the byte offset of the starting position of the written data relative to the beginning of the file; length is the length in bytes, representing the length of the written data; data range represents the byte range of the written data in the file):

[0077] "data10: offset=0, length=256KB, data range [0, 256KB); "

[0078] data12: offset = 300KB, length = 212KB, data range [400KB, 512KB);

[0079] data15: offset = 600KB, length = 100KB, data range [600KB, 700KB);

[0080] data17: offset = 800KB, length = 100KB, data range [800KB, 900KB);

[0081] data11: offset = 128KB, length = 300KB, data range [128KB, 428KB).

[0082] As can be seen, data11 is the last data to be written, and the data range of some data in data11 overlaps with that of data10 and data12.

[0083] In this embodiment, the distributed file system uses a Write-Ahead Log (WAL) mechanism to manage files, recording the write operations as logs and storing the logs in a cache space (such as a linear space, LS). The logs in the cache space are shown below (it should be noted that this is only an example; in some scenarios, logs may include other information besides the original data, such as basic data information, etc., which is not limited in this application):

[0084] "log10: LS offset=0, length=256KB;

[0085] log12: LS offset=256KB, length=212KB;

[0086] log15: LS offset=468KB, length=100KB;

[0087] log17: LS offset=568KB, length=100KB;

[0088] log11: LS offset=668KB, length=300KB”.

[0089] As can be seen, the logs corresponding to the above write operations occupy 968KB of storage space in the cache space. It should be understood that after each write operation's log is successfully written to the cache space, the distributed file system returns a write success message to the client. That is, this IO operation does not need to wait for data persistence, thus effectively improving data write speed. Furthermore, the cache space in this application can be implemented using non-volatile storage media, ensuring that data is not lost.

[0090] Based on storing the logs corresponding to the above write operations in the cache space, this application provides a redirection method based on the WAL mechanism: Based on the amount of log data in the cache space, the logs in the cache space are persistently stored. If the amount of data in the first log in the cache space does not meet a first condition, the first log is redirected to release its initial position in the cache space; if the amount of data in the second log in the cache space meets the first condition, the second log is persistently stored. Illustratively, the first condition may be, for example, that the amount of log data reaches a first threshold. The first threshold can be set according to business needs, and this application does not limit it. The first log and the second log refer to any log in the cache space. The above process describes how, during the persistent storage of logs in the cache space, the distributed file system persists a log entry only if its data volume meets the first condition. Otherwise, it redirects the entry to release its initial position in the cache space, allowing the distributed file system to continue processing the next log entry. This breaks the design of WAL mechanisms in related technologies, which require sequential flushing and persistence along a linear path. It provides the possibility of aggregating discrete data over a longer time span. Furthermore, since the persistent storage process is based on the log data volume, logs whose data volume meets the first condition are persisted. In other words, the amount of log data written to the persistent space meets the first condition, thus avoiding fragmentation of the persistent space and making it applicable to any type of file.

[0091] For example, continuing to refer to Figure 4, taking the first condition of the log data volume reaching 2MB as an example, the distributed file system controls the cursor of the cache space to move in a specified direction. The cursor's starting position is LS offset = 0, that is, the cursor points to log10, and the data volume of log10 is 256KB. Since the data volume of log10 does not meet the first condition, log10 is redirected to release the initial position of the log in the cache space. For example, log10 can be stored at the end of the cache space in an append-only manner; or log10 can be merged with logs in the cache space that can form a continuous log with log10 to obtain a new log (here referring to a new log relative to log10), and the new log can be stored at the end of the cache space in an append-only manner, and so on. For the implementation method of redirection, please refer to the following method embodiments. Figure 4 takes the example of a cache space containing logs that can form a continuous log with log10. Log10, log11, and log12 are merged. During the merging process, since the data range of some data in data11 overlaps with that of data10 and data12, the overlapping parts can be deduplicated to obtain new data "date10-11-12: offset=0, length=512KB, data range [0, 512KB)", which in turn yields a new log "newlog10-11-12: LS offset=12MB, length=512KB". Newlog10-11-12 is then appended to the end of the cache space, releasing the initial position of log10 in the cache space.

[0092] Additionally, during the aforementioned redirection process, the distributed file system can mark the log status of log11 and log12 as "processed." When the cursor in the cache space moves to log11 or log12, the initial position of log11 or log12 in the cache space is released based on its log status. Next, the distributed file system controls the cursor in the cache space to continue moving in the specified direction. When the cursor points to log15 or log17, since neither log15 nor log17 satisfies the first condition, redirection is performed in the same manner as described above to release the initial positions of log15 and log17 in the cache space. For example, log15 and log17 can be appended to the end of the cache space, resulting in "newlog15: LS offset = 12.5MB, length = 100KB; newlog17: LS offset = 12.6MB, length = 100KB".

[0093] After the above redirection, the distributed file system stores multiple new log entries "newlog10-11-12, newlog15, and newlog17" in the cache space, releasing the storage space occupied by multiple old log entries "log10, log12, log15, log17, and log11", totaling 968KB of freed storage space in the cache space. The storage space occupied by the new log entries is 712KB. It should be understood that as file operations continue to be performed, new log entries will be continuously written to the cache space. When the cursor in the cache space points to newlog10-11-12, newlog15, or newlog17, the redirection method described above can continue to be executed until the data volume of a log entry meets the first condition, at which point that log entry is persisted.

[0094] As can be seen, the WAL-based redirection method provided in this application can effectively avoid fragmentation of the persistent space for any type of file. Specifically, when redirecting logs that do not meet the first condition, the log can be stored in the cache space in an append-only manner, or it can be merged with other logs and stored in the cache space in an append-only manner. Thus, when the distributed file system receives an access request for the data corresponding to that log, it can quickly read the corresponding data from the cache space without accessing the persistent space, greatly improving data access performance. For example, in specific business scenarios of high-performance computing (HPC) (such as energy exploration, meteorology, and oceanography), the input-output (IO) model typically exhibits random and disordered access positions within a file, mixed IO of different sizes, high concurrency of multiple jobs, and no obvious characteristic pattern of IO over a period of time. For this type of IO model, the solution provided in this application can improve system access performance while effectively avoiding persistent space fragmentation.

[0095] The following describes in detail the data processing method in the distributed file system provided in this application through several method implementation examples.

[0096] Figure 5 is a flowchart of a data processing method in a distributed file system provided in an embodiment of this application. As shown in Figure 5, the method is applied to a distributed file system. For example, the method is executed by any storage node in the distributed file system. Schematic, the method includes the following steps 501 and 502.

[0097] 501. Distributed file systems store file logs in a cache space, and the logs are used to record operations on the files.

[0098] In this embodiment, "file" refers to any file in the distributed file system. This application does not limit the type of data in the file; it can be text, images, audio, video, etc., and is not limited thereto. Illustratively, in response to a client's operation request for a file, the distributed file system executes the operation indicated by the operation request in memory, records the operation as a log, and stores the log in a cache space. Here, memory refers to the memory of the storage node, which can be either random access memory (RAM) or read-only memory (ROM). For example, RAM can be dynamic random access memory (DRAM) or SCM. Of course, memory can also include other types of RAM, and this application does not limit this. Furthermore, regarding operation requests and operations on files, please refer to the foregoing description, and it will not be repeated here.

[0099] 502. The distributed file system persists the logs in the cache space based on the amount of log data in the cache space.

[0100] In this embodiment, the cache space can be a linear space (LS). The distributed file system persists the logs in the cache space based on whether the amount of log data in the cache space meets a first condition. Indicatively, the first condition refers to the amount of log data reaching a first threshold. This first threshold is a preset threshold that can be set according to business needs, and this application does not limit it.

[0101] In some embodiments, the first condition refers to the log data volume reaching the striping threshold of the distributed file system. It should be understood that, based on the aforementioned content in Figure 2, in a distributed file system employing the EC mechanism, setting the striping threshold determines when the system performs striping operations. The striping threshold indicates the granularity at which the distributed file system performs striping operations. By setting the first condition to the data volume reaching the striping threshold, and persistently storing logs with data volumes reaching the striping threshold, the utilization rate of persistent storage space can be significantly improved, enhancing data access efficiency. It should be understood that, based on the aforementioned introduction to the EC mechanism, for a distributed file system employing the EC mechanism, data read or write operations are performed at the striping threshold as the smallest granularity to ensure data reliability. In this application, persistently storing logs with data volumes reaching the striping threshold avoids the system persistently storing too much invalid data. This process can also be understood as a striping affinity-based redirection method, improving system access performance by affixing affinity to the persistent storage space of the distributed file system.

[0102] Of course, the first threshold indicated by the first condition can also be determined in other ways. For example, if the distributed file system uses a replication mechanism for persistent storage, the first condition can be set to the amount of log data reaching the replication threshold of the distributed file system, etc. This application does not limit this.

[0103] Based on the aforementioned content in Figure 4, this application provides a redirection method based on the WAL mechanism: Based on the amount of log data in the cache space, the logs in the cache space are persistently stored. Specifically, if the amount of data in the first log in the cache space does not meet a first condition, the first log is redirected to release its initial position in the cache space; if the amount of data in the second log in the cache space meets the first condition, the second log is persistently stored. The implementation of this process is described below using the two scenarios shown in steps A and B as examples.

[0104] Step A: Determine that the amount of data in the first log in the cache space does not meet the first condition, and redirect the first log to release its initial position in the cache space.

[0105] Here, the first log refers to any log entry in the cache space. In some embodiments, the cache space is a linear space. The distributed file system controls the cursor of the cache space to move in a specified direction. When the cursor of the cache space points to the first log, it is determined whether the data volume of the first log satisfies a first condition. If not, the first log is redirected to release its initial position in the cache space. The distributed file system's redirection of the first log to release its initial position involves the following situations:

[0106] Scenario 1: Redirect the first log to the target location in the cache space and release the first log from its initial location in the cache space.

[0107] In this process, the target location differs from the initial location; that is, the first log is redirected to a different location within the cache space. By redirecting the first log to the target location within the cache space, when the distributed file system receives an access request for the data corresponding to the first log, it can quickly read the corresponding data from the cache space without accessing the persistent space, greatly improving data access performance. This process can be illustrated in the following ways:

[0108] Method 1: The distributed file system stores the first log file at the end of the cache space using an append-only method, releasing the initial position of the first log file in the cache space. In Method 1, the position occupied by the newly written first log file in the cache space is also the target position.

[0109] In this way, when the distributed file system receives an access request for the data corresponding to the first log, it can quickly read the corresponding data from the cache space without accessing the persistent space, which greatly improves data access performance.

[0110] Method 2: The distributed file system merges the first and eighth logs into a ninth log. The ninth log is then appended to the end of the cache space, releasing the first log's initial position in the cache. The distributed file system marks the eighth log's status to indicate it has been processed. This allows subsequent persistent storage processes to access the eighth log and release its position in the cache space based on its status. The position occupied by the ninth log in the cache space is the target position.

[0111] In Method 2, the ninth log is a continuous log. It should be noted that, in this application, a continuous log refers to a log formed by multiple consecutive logs. For example, logs 1 and 2 are continuous logs. Merging logs 1 and 2 into log 3 results in a continuous log, whose data includes data from both logs 1 and 2. Similarly, in Method 2, the first and eighth logs are continuous logs, or logs capable of forming a continuous log. By merging the first and eighth logs into the ninth log, the discrete data in the cache space is merged, facilitating subsequent persistent storage.

[0112] In some embodiments, the data volume of the ninth log does not meet the first condition. That is, if the data volume of the ninth log obtained after merging the first and eighth logs still does not meet the first condition, the ninth log is stored in the cache space. In this way, logs that do not meet the first condition are kept in the cache space for as long as possible. Thus, when the distributed file system receives an access request for the data corresponding to that log, it can quickly read the corresponding data from the cache space without accessing the persistent storage space, greatly improving data access performance. Alternatively, if the data volume of the ninth log obtained after merging meets the first condition, the distributed file system can persistently store the ninth log, which will not be elaborated further here.

[0113] Furthermore, in the process of merging the first and eighth logs into the ninth log, if there is overlap between the first and eighth logs, the overlapping portions are removed before merging into the ninth log. The method for removing duplicates, such as retaining new data and overwriting old data, is not limited in this application; in practical applications, it can be configured according to the application scenario of the distributed file system. By removing duplicates during the redirection process, cache space usage can be saved, improving the utilization rate of storage resources.

[0114] The above methods 1 and 2 describe how to redirect the first log to a target location in the cache space. In practical applications, the distributed file system can use any of these methods to implement redirection. In other embodiments, the distributed file system can determine whether to use method 1 or method 2 by querying the management information of the cache space, where the management information indicates whether different logs of the file can form a continuous log. Illustratively, before redirecting the first log to the target location, the distributed file system queries the management information of the cache space; if the management information indicates that there is no log in the cache space that can form a continuous log with the first log, the first log is appended to the end of the cache space; if the management information indicates that there is a log in the cache space that can form a continuous log with the first log, the first log and the eighth log are merged into the ninth log.

[0115] The aforementioned management information can be stored in the memory of the storage node, and this application does not limit the form of the management information. For example, management can use a hash or tree data structure to record different data in the file, such as recording the offset and length of different data in the file. Based on this, the distributed file system can determine whether different data in the file are continuous, that is, determine whether there are continuous logs in the cache space. The above process is as follows: when it is determined that the amount of data in the first log does not meet the first condition, the management information of the cache space is queried. If it is determined that the management information indicates that there are no logs in the cache space that can form continuous logs with the first log, then the first log is directly rewritten to the cache space in an append-only manner. If it is determined that the management information indicates that there are logs in the cache space that can form continuous logs with the first log, then the first log and the logs that can form continuous logs are merged, and the merged continuous log is written to the cache space in an append-only manner.

[0116] Scenario 2: Merge the first log and the sixth log into the seventh log, redirect the first log to another log outside the first log, release the initial position, and persist the seventh log.

[0117] In this case, the data volume of the seventh log satisfies the first condition, and the seventh log is a continuous log. It should be understood that, similar to the process of merging the first and eighth logs into the ninth log in Case 1, in Case 2, the first and sixth logs are continuous logs, or logs capable of forming a continuous log. By merging the first and sixth logs into the seventh log, whose data volume satisfies the first condition, the discrete data in the cache space is merged. Moreover, since the merged log satisfies the first condition, the seventh log can be persistently stored. This process is equivalent to redirecting the first log to another log outside the first log, thereby releasing the initial position of the first log in the cache space.

[0118] In addition, in scenario two, the distributed file system can also query the management information of the cache space to determine that the management information indicates that there is a log in the cache space that can form a continuous log with the first log. The first log and the sixth log are merged into the seventh log to redirect the first log to other logs other than the first log. This process is the same as the relevant content in scenario one above, so it will not be described again.

[0119] Step B: Determine that the amount of data in the second log in the cache space meets the first condition, and then persist the second log.

[0120] Here, the second log refers to any log entry in the cache space. In some embodiments, the cache space is a linear space, and the distributed file system controls the cursor of the cache space to move in a specified direction. When the cursor of the cache space points to the second log, it is determined whether the amount of data in the second log meets the first condition. If it does, the second log is persisted.

[0121] Through steps 501 and 502 above, a data processing method in a distributed file system is introduced. This method involves storing file logs in a cache space, and then persisting the logs based on the amount of data in the cache space. During this process, if the amount of data in a log entry meets a first condition, that log entry is persisted; otherwise, it is redirected to release its initial position in the cache space, allowing the distributed file system to continue processing the next log entry. This breaks the design of related technologies where the WAL mechanism must advance sequentially in a linear space for persistence, providing the possibility of aggregating discrete data over a longer time span. Furthermore, since the persistence process is based on the amount of data in the logs, logs whose data amount meets the first condition are persisted. In other words, the amount of data written to the persistent space meets the first condition, thus avoiding fragmentation of the persistent space and making it applicable to any type of file.

[0122] Furthermore, considering the limited storage space of the cache, this application also provides a persistent storage mechanism based on dispersion, which can prevent discrete data from occupying the cache space for a long time. For example, under certain IO models, even if all logs are repeatedly redirected until the cache space is exhausted, the first condition cannot be met. In this case, dispersion is used to measure the proportion of logs in the cache space that still cannot meet the first condition after repeated redirection, thereby determining whether it is necessary to persist some logs that do not meet the first condition and release the cache space in a timely manner. This persistent storage mechanism is described below.

[0123] Schematic illustration: The distributed file system persists the logs of files in the cache space to release the cache space based on the dispersion of logs in the cache space. Here, dispersion indicates the number of logs in the cache space that cannot form contiguous logs with other logs. This application does not limit the method for determining dispersion; for example, the dispersion of logs in the cache space can be determined using the following formula (1): X=(T×M+K×N) / A (1)

[0124] In formula (1), X represents the dispersion of logs in the cache space, T represents the dispersion coefficient of logs that have undergone one redirection (e.g., log a) (set to 0.8), M represents the storage space occupied by this part of log a in the cache space, K represents the dispersion coefficient of logs that have undergone two or more redirections (e.g., log b) (set to 1), N represents the storage space occupied by this part of log b in the cache space, and A represents the total storage space of the cache space. It should be noted that the dispersion determination method shown in formula (1) is only an example and does not constitute a limitation of this application. In practical applications, for example, the number of logs that have undergone redirection in the cache space can also be used as the dispersion, or the ratio of the number of logs that have undergone redirection in the cache space to the total number of logs in the cache space can be used as the dispersion, and so on.

[0125] In some embodiments, the distributed file system determines that the dispersion of logs in the cache space meets a second condition, and then persists the third logs of the files in the cache space. The second condition is, for example, that the dispersion reaches a second threshold. The second threshold can be set according to business needs, and this application does not limit it. Taking the above formula (1) as an example, the second threshold is, for example, 90%.

[0126] The aforementioned third log can be any log in the cache space. That is, when there is a large amount of discrete data in the cache space, a certain log can be forcibly persisted to release the cache space. In some embodiments, the third log is a log in the cache space whose data volume meets a third condition, such as the data volume being greater than a third threshold and the third threshold being less than a first threshold. That is, persisting a log with a relatively large amount of data in the cache space can minimize the fragmentation of the persistent space. In still other embodiments, the third log refers to a log that has undergone redirection. That is, when there is a large amount of discrete data in the cache space, a certain log that has undergone redirection can be forcibly persisted. Since the third log has undergone redirection, there is a high probability that there will be no logs in the cache space that form a continuous log with the third log in the short term. Persisting the third log releases the cache space in a timely manner.

[0127] In some embodiments, when the dispersion of logs in the cache space meets the second condition and there are multiple third logs, the distributed file system determines a fourth log that can form a continuous log with the multiple third logs; reads the fourth log from the persistent storage space, merges the multiple third logs and the read fourth log into a fifth log, and persists the fifth log. The fifth log is a continuous log. Here, the third log refers to logs that have undergone redirection. The distributed file system can determine the fourth log by querying the management information of the cache space, which will not be elaborated here. This process means that when there is a large amount of discrete data in the cache space and multiple third logs awaiting persistent storage, since the third logs have undergone redirection, it indicates that there may be continuous logs that have already been persisted. Therefore, these logs can be read from the persistent storage space, merged with the third logs to obtain the fifth log, and persisted. In this way, the discrete logs in the cache space are concatenated into a continuous log before persistent storage, avoiding fragmentation of the persistent space and improving the access performance of the persistent space.

[0128] Furthermore, after the distributed file system reads the fourth log from the persistent storage space, it reclaims the fourth log in the persistent storage space. Illustratively, reclamation refers to garbage collection; this application does not limit the garbage collection method used by the system, but can refer to the foregoing description, which will not be repeated here. By reclamating the fourth log in the persistent space, the persistent space is released in a timely manner, improving disk space utilization.

[0129] In some scenarios, if a distributed file system determines that there are no consecutive logs between two third-level logs, it will pad the missing data range between the two third-level logs with zeros to obtain a consecutive log. In other words, if there are no consecutive logs between two third-level logs, it means that the missing data range between them has not yet been operated on, so zeros are padded to ensure that the persistently stored logs are consecutive.

[0130] In other embodiments, the data volume of the fifth log satisfies the first condition. That is, during persistent storage, whether the data volume of the fifth log satisfies the first condition is considered to further improve system access performance. Illustratively, the distributed file system determines the fourth log based on a first threshold indicated by the first condition so that the data volume of the fifth log satisfies the first condition. For example, taking a first threshold of 2MB as an example, if the total data volume of multiple third logs is 1.5MB, the system can determine a fourth log with a data volume less than or equal to 0.5MB, ensuring that the fifth log obtained by merging the fourth log and multiple third logs satisfies the first condition.

[0131] Referring to Figure 6, the above-described persistent storage mechanism based on discreteness will be illustrated below. Figure 6 is a schematic diagram of a persistent storage mechanism based on discreteness provided in an embodiment of this application. As shown in Figure 6, the cache space includes multiple logs. When the discreteness of the logs in the cache space meets the second condition, log10, log12, and log14 are treated as multiple third logs and prepared for persistent storage. For example, the data corresponding to log10, log12, and log14 are as follows:

[0132] "data10: offset=1MB, length=256KB, data range [1MB, 1.25MB);

[0133] data12: offset = 1.5MB, length = 256KB, data range [1.5MB, 1.75MB);

[0134] data14: offset = 2.5MB, length = 512KB, data range [2.5MB, 3MB).

[0135] As can be seen, the data ranges corresponding to log10, log12, and log14 are not continuous. Based on this, and combined with the first threshold indicated by the first condition, the range of data to be read from the persistent storage space is determined. For example, if the first threshold is 2MB, to satisfy the first condition, offset = 1MB, length = 2MB, and the data range is [1MB, 3MB). Based on this data range, it is determined that two segments of data are missing, namely [1.25MB, 1.5MB) and [1.75MB, 2.5MB). Based on this, the distributed file system reads this part of the data from the persistent storage space. Here, the read data is referred to as data11 and data13, that is, the data corresponding to log11 and log13, or the missing data. After successful reading, data10, data11, data12, data13, and data14 are merged in memory to obtain a whole-segment data with a data range of [1MB, 3MB), that is, the fifth log with the data volume satisfying the first condition, and the fifth log is stored persistently.

[0136] In addition, the distributed file system updates the file's metadata and marks the old data11 and old data13 as garbage, which are then reclaimed by the GC mechanism. Furthermore, data10, data11, data12, data13, and data14 are written to the persistent medium in whole stripes and distributed contiguously on the medium, avoiding fragmentation and improving access speed.

[0137] Figure 7 is a flowchart of another data processing method in a distributed file system provided in an embodiment of this application. As shown in Figure 7, the method is applied to a distributed file system. For example, the method is executed by any storage node in the distributed file system. Schematic, the method includes the following steps 701 to 706.

[0138] 701. Distributed file systems store file logs in a cache space, and the logs are used to record operations on the files.

[0139] Step 701 is the same as step 501 in the embodiment shown in Figure 5 above, so it will not be described again.

[0140] 702. The distributed file system determines that the amount of data in the first log in the cache space does not meet the first condition.

[0141] Here, the first log refers to any log entry in the cache space. This embodiment describes how the distributed file system handles the first log when its data volume does not meet the first condition.

[0142] 703. Distributed file system query cache space management information, which is used to indicate whether different logs of a file can form a continuous log.

[0143] The meaning of the management information is as described in step A of the embodiment shown in Figure 5 above, and will not be repeated here.

[0144] Steps 702 and 703 above refer to the distributed file system querying the management information of the cache space when it determines that the data volume of the first log does not meet the first condition. Based on this, if the distributed file system determines that the management information indicates that there is no log in the cache space that can form a continuous log with the first log, then it executes step 704, directly rewriting the first log into the cache space in an append-only manner; if the distributed file system determines that the management information indicates that there is a log in the cache space that can form a continuous log with the first log, then it executes step 705, merging the first log and the log that can form a continuous log, and writing the merged continuous log into the cache space in an append-only manner.

[0145] 704. If the distributed file system determines that there is no log in the cache space that can form a continuous log with the first log, it will store the first log at the end of the cache space in an append-only manner and release the initial position of the first log in the cache space.

[0146] Step 704 refers to step A in the embodiment shown in Figure 5 above, and will not be described again.

[0147] 705. The distributed file system determines that there is a log in the cache space that can form a continuous log with the first log, determines that the first log and the eighth log are merged into the ninth log, determines that the ninth log is stored at the end of the cache space in an append-only manner, and determines that the first log is released from its initial position in the cache space.

[0148] Step 705 refers to step A in the embodiment shown in Figure 5 above, and will not be repeated here.

[0149] 706. If the distributed file system determines that the dispersion of logs in the cache space meets the second condition, it will persist the third log of the file in the cache space. The dispersion is used to indicate the number of logs in the cache space that cannot form a continuous log with other logs.

[0150] Step 706 refers to the discrete-degree-based persistent storage mechanism in the embodiment shown in Figure 5 above, and will not be repeated here.

[0151] As illustrated in Figure 7, the distributed file system stores file logs in a cache space. Logs are then persistently stored based on whether the amount of data in the cache space meets certain conditions. During this process, if the amount of data in a log entry does not meet the conditions, the cache space management information is queried to redirect the log entry, releasing its initial position in the cache space. This allows the distributed file system to continue processing the next log entry, providing the possibility of aggregating discrete data over a longer time span. This method is applicable to any type of file. Furthermore, since the persistent storage process is based on the amount of data in the logs, only logs that meet the conditions are persistently stored, thus avoiding fragmentation of the persistent storage space and improving system access performance. Additionally, by providing a persistence storage mechanism based on discreteness, it is possible to prevent discrete data from occupying the cache space for extended periods.

[0152] Figure 8 is a schematic diagram of the structure of a data processing device in a distributed file system provided in an embodiment of this application. As shown in Figure 8, the device includes a first storage module 801 and a second storage module 802.

[0153] The first storage module 801 is used to store the logs of files in the distributed file system into the cache space. The logs are used to record operations on the files.

[0154] The second storage module 802 is used to persistently store the logs in the cache space based on the amount of log data in the cache space;

[0155] Specifically, if the amount of data in the first log in the cache space does not meet the first condition, the first log is redirected to release its initial position in the cache space.

[0156] In one possible implementation, the second storage module 802 is used for:

[0157] If the amount of data in the second log in the cache space meets the first condition, the second log will be persisted.

[0158] In one possible implementation, the device further includes:

[0159] The third storage module is used to determine if the dispersion of logs in the cache space meets the second condition, and to persistently store the third logs of the files in the cache space. The dispersion is used to indicate the number of logs in the cache space that cannot form continuous logs with other logs.

[0160] In one possible implementation, the third log refers to the log that has been redirected.

[0161] In one possible implementation, the third storage module is used for:

[0162] If the dispersion of logs in the cache space satisfies the second condition and there are multiple third logs, determine the fourth log that can form a continuous log with multiple third logs;

[0163] The fourth log is read from the persistent storage space. Multiple third logs and the read fourth log are merged into the fifth log. The fifth log is then stored persistently. The fifth log is a continuous log.

[0164] In one possible implementation, the second storage module 802 is used for:

[0165] If the data volume of the first log entry does not meet the first condition, redirect the first log entry to the target location in the cache space and release the initial location. The target location is different from the initial location. Alternatively...

[0166] If the data volume of the first log does not meet the first condition, the first log and the sixth log are merged into the seventh log to redirect the first log to another log other than the first log. The initial position is released and the seventh log is persisted. The data volume of the seventh log meets the first condition and the seventh log is a continuous log.

[0167] In one possible implementation, the second storage module 802 is used for:

[0168] Store the first log entry at the end of the cache space in an append-only manner; or,

[0169] The first and eighth logs are merged into the ninth log, and the ninth log is stored at the end of the cache space in an append-only manner. The ninth log is a continuous log, and the amount of data in the ninth log does not meet the first condition.

[0170] In one possible implementation, the device further includes:

[0171] The query module is used to query the management information of the cache space. The management information is used to indicate whether different logs of a file can form a continuous log.

[0172] The second storage module 802 is used for any of the following:

[0173] If the management information indicates that there is no log in the cache space that can form a continuous log with the first log, the first log is stored at the end of the cache space by appending it.

[0174] If the management information indicates that there is a log in the cache space that can form a continuous log with the first log, merge the first log and the eighth log into the ninth log.

[0175] In one possible implementation, the first condition is that the amount of log data reaches the striping threshold of the distributed file system, which is used to indicate the granularity at which the distributed file system performs striping operations.

[0176] Both the first storage module 801 and the second storage module 802 can be implemented in software or in hardware. For example, the implementation of the first storage module 801 will be described below. Similarly, the implementation of the second storage module 802 can refer to the implementation of the first storage module 801.

[0177] As an example of a software functional unit, the first storage module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first storage module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0178] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0179] As an example of a hardware functional unit, the first storage module 801 may include at least one computing device, such as a server. Alternatively, the first storage module 801 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0180] The multiple computing devices included in the first storage module 801 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first storage module 801 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first storage module 801 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0181] It should be noted that, in other embodiments, the first storage module 801 can be used to execute any step in the data processing method in the distributed file system, and the second storage module 802 can be used to execute any step in the data processing method in the distributed file system. The steps implemented by the first storage module 801 and the second storage module 802 can be specified as needed. By implementing different steps in the data processing method in the distributed file system through the first storage module 801 and the second storage module 802 respectively, all functions of the data processing device in the distributed file system can be realized.

[0182] This application also provides a storage node 900. Figure 9 is a schematic diagram of the structure of a storage node provided in an embodiment of this application. As shown in Figure 9, the storage node 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The storage node 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the storage node 900.

[0183] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 904 can include pathways for transmitting information between various components of storage node 900 (e.g., memory 906, processor 904, communication interface 908).

[0184] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0185] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, NVDIMM, SCM, hard disk drive (HDD), or solid state drive (SSD).

[0186] The memory 906 stores executable program code, and the processor 904 executes this executable program code to implement the functions of the aforementioned first storage module and second storage module, thereby realizing the data processing method in the distributed file system. That is, the memory 906 stores instructions for executing the data processing method in the distributed file system.

[0187] Alternatively, the memory 906 stores executable code, which the processor 904 executes to implement the functions of the data processing device in the aforementioned distributed file system, thereby implementing the data processing method in the distributed file system. That is, the memory 906 stores instructions for executing the data processing method in the distributed file system.

[0188] The communication interface 903 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the storage node 900 and other devices or communication networks.

[0189] This application also provides a storage cluster. Figure 10 is a schematic diagram of a storage cluster provided in this application embodiment. As shown in Figure 10, the storage cluster includes at least one storage node 900. The memory 906 in one or more storage nodes 900 in the storage cluster may store the same instructions for executing data processing methods in a distributed file system.

[0190] In some possible implementations, the memory 906 of one or more storage nodes 900 in the storage cluster may also store partial instructions for executing data processing methods in the distributed file system. In other words, a combination of one or more storage nodes 900 can jointly execute instructions for executing data processing methods in the distributed file system.

[0191] It should be noted that the memory 906 in different storage nodes 900 in the storage cluster can store different instructions, which are used to execute some functions of the data processing device in the distributed file system. That is, the instructions stored in the memory 906 in different storage nodes 900 can realize the functions of one or more of the aforementioned first storage module 801, second storage module 802, third storage module and query module.

[0192] It should be understood that the function of the storage node 900 shown in Figure 10 can also be performed by multiple storage nodes 900.

[0193] In some possible implementations, one or more storage nodes in a storage cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. Figure 11 is a schematic diagram of a possible implementation of a storage cluster provided by an embodiment of this application. As shown in Figure 11, storage node 900A and storage node 900B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each storage node. In this type of possible implementation, the memory 906 in storage node 900A stores instructions for performing the functions of the first and second storage modules. Simultaneously, the memory 906 in storage node 900B stores instructions for performing the functions of the query module.

[0194] This application also provides a computer program product containing instructions. This computer program product can be software or program products containing instructions capable of running on a storage cluster or stored on any available medium. When the computer program product runs on the storage cluster, it causes the storage cluster to execute a distributed file system data processing method.

[0195] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a storage cluster can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the storage cluster to execute a data processing method of a distributed file system.

[0196] Those skilled in the art will recognize that the method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0200] Furthermore, the units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or software.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, or computing device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first log can be referred to as a second log, and similarly, a second log can be referred to as a first log. Both a first log and a second log can be logs, and in some cases, they can be separate and distinct logs.

[0203] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. The terms "system" and "network" are often used interchangeably.

[0204] It should also be understood that the term "if" can be interpreted as meaning "when" or "upon" or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" can be interpreted as meaning "when determination..." or "in response to determination..." or "when detection [the stated condition or event]" or "in response to detection [the stated condition or event]."

[0205] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0206] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer program instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0207] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs), or semiconductor media (e.g., solid-state drives)).

[0208] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0209] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method in a distributed file system, characterized in that, The method includes: The logs of files in the distributed file system are stored in the cache space, and the logs are used to record operations on the files. Based on the amount of log data in the cache space, the logs in the cache space are persistently stored. Specifically, if the amount of data in the first log in the cache space does not meet the first condition, the first log is redirected to release its initial position in the cache space.

2. The method according to claim 1, characterized in that, The step of persistently storing the logs in the cache space based on the amount of log data in the cache space includes: If the amount of data in the second log in the cache space meets the first condition, the second log is persisted.

3. The method according to claim 1 or 2, characterized in that, The method further includes: If the dispersion of the logs in the cache space satisfies the second condition, the third log of the file in the cache space is persistently stored. The dispersion is used to indicate the number of logs in the cache space that cannot form continuous logs with other logs.

4. The method according to claim 3, characterized in that, The third log refers to the log that has already undergone redirection.

5. The method according to claim 3 or 4, characterized in that, The step of determining that the dispersion of the logs in the cache space meets the second condition, and then persistently storing the third logs of the file in the cache space, includes: If the dispersion of logs in the cache space satisfies the second condition and there are multiple third logs, a fourth log that can form a continuous log with multiple third logs is determined. The fourth log is read from the persistent storage space, and the multiple third logs and the read fourth log are merged into a fifth log. The fifth log is then stored persistently, and the fifth log is a continuous log.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining that the amount of data in the first log in the cache space does not meet the first condition, and redirecting the first log to release its initial position in the cache space, includes: If the data volume of the first log does not meet the first condition, the first log is redirected to a target location in the cache space, and the initial location is released. The target location is different from the initial location; or... If the data volume of the first log does not meet the first condition, the first log and the sixth log are merged into a seventh log to redirect the first log to another log other than the first log. The initial position is released and the seventh log is persistently stored. The data volume of the seventh log meets the first condition, and the seventh log is a continuous log.

7. The method according to claim 6, characterized in that, The step of determining that the data volume of the first log does not meet the first condition, redirecting the first log to the target location in the cache space, and releasing the initial location includes: The first log is appended to the end of the cache space; or, The first log and the eighth log are merged into the ninth log, and the ninth log is stored at the end of the cache space in an append-only manner. The ninth log is a continuous log, and the amount of data in the ninth log does not meet the first condition.

8. The method according to claim 7, characterized in that, The method further includes: Query the management information of the cache space, which indicates whether different logs of the file can form a continuous log; The step of storing the first log at the end of the cache space in an append-only manner includes: determining that the management information indicates that there is no log in the cache space that can form a continuous log with the first log, and storing the first log at the end of the cache space in an append-only manner; The step of merging the first log and the eighth log into the ninth log includes: determining that the management information indicates that there is a log in the cache space that can form a continuous log with the first log, and merging the first log and the eighth log into the ninth log.

9. The method according to any one of claims 1 to 8, characterized in that, The first condition refers to the log data volume reaching the striping threshold of the distributed file system, which is used to indicate the granularity of the striping operation performed by the distributed file system.

10. A data processing device in a distributed file system, characterized in that, The device includes: The first storage module is used to store the logs of files in the distributed file system into the cache space, and the logs are used to record operations on the files. The second storage module is used to persistently store the logs in the cache space based on the amount of log data in the cache space. Specifically, if the amount of data in the first log in the cache space does not meet the first condition, the first log is redirected to release its initial position in the cache space.

11. The apparatus according to claim 9, characterized in that, The second storage module is used for: If the amount of data in the second log in the cache space meets the first condition, the second log is persisted.

12. The apparatus according to claim 10 or 11, characterized in that, The device further includes: The third storage module is used to determine that the dispersion of the logs in the cache space meets the second condition, and to persistently store the third logs of the file in the cache space. The dispersion is used to indicate the number of logs in the cache space that cannot form continuous logs with other logs.

13. The apparatus according to claim 12, characterized in that, The third log refers to the log that has already undergone redirection.

14. The apparatus according to claim 12 or 13, characterized in that, The third storage module is used for: If the dispersion of logs in the cache space satisfies the second condition and there are multiple third logs, a fourth log that can form a continuous log with multiple third logs is determined. The fourth log is read from the persistent storage space, and the multiple third logs and the read fourth log are merged into a fifth log. The fifth log is then stored persistently, and the fifth log is a continuous log.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The second storage module is used for: If it is determined that the data volume of the first log does not meet the first condition, the first log is redirected to the target location in the cache space, and the initial location is released. The target location is different from the initial location. or, If the data volume of the first log does not meet the first condition, the first log and the sixth log are merged into a seventh log to redirect the first log to another log other than the first log. The initial position is released and the seventh log is persistently stored. The data volume of the seventh log meets the first condition, and the seventh log is a continuous log.

16. The apparatus according to claim 15, characterized in that, The second storage module is used for: The first log is appended to the end of the cache space; or, The first log and the eighth log are merged into the ninth log, and the ninth log is stored at the end of the cache space in an append-only manner. The ninth log is a continuous log, and the amount of data in the ninth log does not meet the first condition.

17. The apparatus according to claim 16, characterized in that, The device further includes: The query module is used to query the management information of the cache space, and the management information is used to indicate whether different logs of the file can form continuous logs; The second storage module is used for any of the following: If the management information indicates that there is no log in the cache space that can form a continuous log with the first log, the first log is stored at the end of the cache space in an append-only manner. If the management information indicates that there is a log in the cache space that can form a continuous log with the first log, then merge the first log and the eighth log into the ninth log.

18. The apparatus according to any one of claims 10 to 17, characterized in that, The first condition refers to the log data volume reaching the striping threshold of the distributed file system, which is used to indicate the granularity of the striping operation performed by the distributed file system.

19. A storage cluster, characterized in that, The storage cluster includes at least one storage node, each storage node including a processor and a memory, wherein the processor of the at least one storage node is used to execute instructions stored in the memory of the at least one storage node to cause the storage cluster to perform the data processing method in the distributed file system as described in any one of claims 1 to 9.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the storage cluster, the storage cluster performs the data processing method in the distributed file system as described in any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a storage cluster, enable the storage cluster to perform a data processing method in a distributed file system as described in any one of claims 1 to 9.

Images