Distributed file writing method, device, computer device and system
By configuring input/output nodes for each page and performing data aggregation and writing, the problem of low efficiency in concurrent writing across multiple nodes in distributed file systems is solved, achieving more efficient IO operations and improved storage system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DAWNING INFORMATION IND (BEIJING) CO LTD
- Filing Date
- 2022-12-27
- Publication Date
- 2026-06-02
AI Technical Summary
Existing distributed file systems are inefficient when multiple computing nodes concurrently write to the same page. This is mainly due to the serialization of write operations caused by write permission contention, and the need for frequent disk reads and writes for non-page aligned write requests, resulting in long IO response times and a large number of IO operations.
Configure corresponding input and output nodes for each page, receive write requests from the compute nodes, aggregate the data, and write the aggregated data to the disk in a unified manner to reduce the number of disk read and write operations and avoid write permission competition.
By aggregating and unifying data writing, the efficiency of concurrent writing of the same page by multiple computing nodes is effectively improved, the number of disk I/O operations is reduced, and the performance of the storage system is fully utilized.
Smart Images
Figure CN116049129B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed file technology, and in particular to a distributed file writing method, apparatus, computer device, and system. Background Technology
[0002] In a distributed file system, multiple compute nodes can concurrently write to the same page of the same file. To ensure data integrity, a write permission management module (also known as a distributed lock management module) is typically deployed. This module manages the write permissions of each compute node to each page (also known as a distributed range lock). For a given page, only one node is granted write permission to that page at any given time. Furthermore, when managing memory, the system divides memory into multiple pages, with page sizes ranging from 4KB to 16KB.
[0003] Specifically, when compute node A performs a write operation on page A of a file, it first needs to obtain write permission for page A of that file from the write permission management module. After obtaining write permission for page A, it modifies the content of page A. Once compute node A confirms the completion of the modification, the write permission management module revokes compute node A's write permission for page A, allowing other compute nodes to obtain write permission for page A and perform read and write operations on page A. Therefore, when multiple compute nodes concurrently write to the same page of the same file, the distributed file system can only execute the write requests of multiple compute nodes serially.
[0004] It is evident that the aforementioned distributed file system is inefficient when performing write operations on compute nodes. Summary of the Invention
[0005] Therefore, it is necessary to provide a distributed file writing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the concurrent writing efficiency of multiple computing nodes in response to the above-mentioned technical problems.
[0006] Firstly, this application provides a distributed file writing method. The method is applied to the input / output nodes of a distributed file system, which further includes multiple computing nodes; the method includes:
[0007] Receive write requests carrying data to be written from each computing node; wherein, the computing node is a computing node in the distributed file system;
[0008] Determine the write address range for each of the data to be written, and determine the page corresponding to each write address range of the data to be written;
[0009] For each page, the data to be written corresponding to each page is aggregated according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page;
[0010] The aggregated data is written to the storage space corresponding to the page on the disk.
[0011] The distributed file system writing method provided in this application effectively reduces the number of disk reads and writes by aggregating the data to be written carried by write requests from multiple computing nodes before writing to the disk. Furthermore, since each page corresponds to an input / output node, and the write permission of each page is assigned to an input / output node, when multiple computing nodes concurrently write to the same page, the write can be uniformly written to the disk by the input / output node corresponding to that page, which effectively improves the writing efficiency of multiple computing nodes concurrently writing to the same page.
[0012] In one embodiment, for each page, the aggregation of the data to be written corresponding to each page, based on the address range of each data to be written corresponding to the page, to obtain the aggregated data of the page, includes:
[0013] For each page, the data to be written corresponding to the page is cached in the target cache queue corresponding to the page;
[0014] If the page meets the first write-back condition, the multiple data to be written are aggregated according to the address range of the multiple data to be written cached in the target cache queue to obtain the aggregated data of the page.
[0015] Input and output nodes use a cache queue to cache the received data to be written. Aggregation is performed when the amount of data to be written for each page reaches a certain amount.
[0016] In one embodiment, for each page, the aggregation of the data to be written corresponding to each page, based on the address range of each data to be written corresponding to the page, to obtain the aggregated data of the page, includes:
[0017] Based on the address range of the data to be written, add the address range of the data to be written to the address range aggregation tree;
[0018] According to the address range aggregation tree, the data to be written corresponding to each address range in the address range aggregation tree is aggregated according to a preset rule;
[0019] When the aggregated data corresponding to the page is aggregated, the aggregated data is cached in the page cache queue;
[0020] The step of writing the aggregated data to the storage space corresponding to the page on the disk includes:
[0021] Read the aggregated data of each page in the page cache queue and write the aggregated data of each page into the storage space corresponding to the page on the disk.
[0022] By organizing the data buffer to be written using an address range aggregation tree, padding reads can be avoided for non-page-aligned write operations, shortening I / O time. Furthermore, for multiple non-page-aligned concurrent write scenarios, page-aligned aggregated data can be generated, avoiding padding reads, reducing I / O counts, and improving overall system bandwidth. In addition, after the data to be written is cached, it is written to disk uniformly, eliminating competition for write permissions between compute nodes and fully leveraging the performance of the storage system.
[0023] In one embodiment, adding the data to be written to the address range aggregation tree according to the address range of the data to be written includes:
[0024] If it is determined that the address ranges corresponding to the pages of the data to be written in the address range aggregation tree do not overlap with the address ranges corresponding to the data to be written, the address ranges of the data to be written are added to the address range aggregation tree.
[0025] When writing data that does not conflict with other data to be written, the input / output nodes directly add the address range corresponding to the new data to be written to the address range tree.
[0026] In one embodiment, adding the data to be written to the address range aggregation tree according to the address range of the data to be written includes:
[0027] If it is determined that the address intervals corresponding to the page to be written in the address interval aggregation tree overlap with the address intervals of the data to be written, delete the overlapping address intervals in the address interval aggregation tree that overlap with the address intervals of the data to be written, delete the data to be written corresponding to the overlapping intervals, and add the address intervals of the data to be written to the address interval aggregation tree.
[0028] When adding the address range corresponding to new data to be written to the address range tree, the input and output nodes dynamically adjust the various address ranges in the address range tree.
[0029] Secondly, this application also provides a distributed file writing method. The method is applied to computing nodes in a distributed file system, which further includes several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; the method includes:
[0030] Determine the address range targeted by the write request, and determine the page corresponding to the address range;
[0031] The write request is sent to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0032] Compute nodes no longer compete with other compute nodes for write permissions on various pages. Instead, they send the corresponding data to be written to the input / output nodes of the corresponding pages, reducing the time loss caused by write permission competition.
[0033] In one embodiment, sending the write request to the input / output node corresponding to the page includes:
[0034] The data to be written corresponding to the write request is cached in the cache queue corresponding to the page;
[0035] If the page meets the preset second write conditions, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page.
[0036] The preliminary aggregated data is sent to the input / output node corresponding to the page.
[0037] When a compute node has multiple write requests, it aggregates the data to be written corresponding to its multiple write requests, obtains preliminary aggregated data, and sends it to the input / output nodes. This reduces the computational load of the input / output nodes for aggregation, reduces communication between the compute node and the input / output nodes, and reduces the number of transmission failures.
[0038] In one embodiment, when it is determined that the page meets the preset second write-back conditions, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page, including:
[0039] If it is determined that the cache queue corresponding to the page has reached a preset length, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page; or
[0040] Upon reaching a preset time, the data corresponding to each write request in the page's queue is aggregated to obtain preliminary aggregated data for the page; or
[0041] When the write task is completed, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page.
[0042] The conditions for data aggregation by compute nodes under various circumstances are set to avoid the compute nodes from not being aggregated for a long time or not being sent to input / output nodes for a long time.
[0043] Thirdly, this application also provides a distributed file system. The distributed file system includes a plurality of computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; wherein:
[0044] The computing node is used to determine the address range targeted by the write request and to determine the page corresponding to the address range; and to send the write request to the input / output node corresponding to the page;
[0045] The input / output node is used to receive write requests carrying data to be written from each computing node; determine the write address range of each data to be written, and determine the page corresponding to each write address range of the data to be written; for each page, aggregate the data to be written corresponding to the page according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page; and write the aggregated data into the storage space corresponding to the page on the disk.
[0046] Fourthly, this application also provides a distributed file writing device. The device is applied to the input / output nodes of a distributed file system, which further includes multiple computing nodes; the device includes:
[0047] A receiving module is used to receive write requests carrying data to be written sent by each computing node; wherein, the computing node is a computing node in the distributed file system;
[0048] The determining module is used to determine the write address range of each of the data to be written, and to determine the page corresponding to each write address range of the data to be written;
[0049] The aggregation module is used to aggregate the data to be written corresponding to each page according to the address range of each data to be written corresponding to the page, so as to obtain the aggregated data of the page.
[0050] The write module is used to write the aggregated data to the storage space corresponding to the page on the disk.
[0051] Fifthly, this application also provides a distributed file writing apparatus. The apparatus is applied to a computing node in a distributed file system, which further includes several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; the apparatus includes:
[0052] The determination module is used to determine the address range targeted by the write request and to determine the page corresponding to the address range;
[0053] The sending module is used to send the write request to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the obtained aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0054] Sixthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0055] Receive write requests carrying data to be written from each computing node; wherein, the computing node is a computing node in the distributed file system;
[0056] Determine the write address range for each of the data to be written, and determine the page corresponding to each write address range of the data to be written;
[0057] For each page, the data to be written corresponding to each page is aggregated according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page;
[0058] The aggregated data is written to the storage space corresponding to the page on the disk.
[0059] Seventhly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0060] Determine the address range targeted by the write request, and determine the page corresponding to the address range;
[0061] The write request is sent to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0062] Eighthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0063] Receive write requests carrying data to be written from each computing node; wherein, the computing node is a computing node in the distributed file system;
[0064] Determine the write address range for each of the data to be written, and determine the page corresponding to each write address range of the data to be written;
[0065] For each page, the data to be written corresponding to each page is aggregated according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page;
[0066] The aggregated data is written to the storage space corresponding to the page on the disk.
[0067] Ninthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0068] Determine the address range targeted by the write request, and determine the page corresponding to the address range;
[0069] The write request is sent to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0070] Tenthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0071] Receive write requests carrying data to be written from each computing node; wherein, the computing node is a computing node in the distributed file system;
[0072] Determine the write address range for each of the data to be written, and determine the page corresponding to each write address range of the data to be written;
[0073] For each page, the data to be written corresponding to each page is aggregated according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page;
[0074] The aggregated data is written to the storage space corresponding to the page on the disk.
[0075] Eleventhly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0076] Determine the address range targeted by the write request, and determine the page corresponding to the address range;
[0077] The write request is sent to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0078] The aforementioned distributed file writing method, apparatus, computer device, system, storage medium, and computer program product configure a corresponding input / output node for each page. When a computing node receives a write request for a page, it sends the data to be written, carried by the write request, to the page's input / output node. When the input / output node receives multiple data items to be written for a page, it aggregates the multiple data items to obtain aggregated data and writes it to the disk.
[0079] By aggregating the data to be written from multiple write requests and then writing it to the disk, the number of disk reads and writes is effectively reduced. Furthermore, since each page corresponds to an input / output node, and the write permission for each page is assigned to an input / output node, when multiple compute nodes write concurrently to the same page, the write can be uniformly written to the disk by the input / output node corresponding to that page, which effectively improves the write efficiency of multiple compute nodes concurrently writing to the same page. Attached Figure Description
[0080] Figure 1 This diagram illustrates the application environment of distributed file systems in related technologies.
[0081] Figure 2 This is an application environment diagram of a distributed file writing method in one embodiment;
[0082] Figure 3This is a flowchart illustrating a distributed file writing method applied to input / output nodes in one embodiment.
[0083] Figure 4 This is a flowchart illustrating a distributed file writing method applied to a computing node in one embodiment;
[0084] Figure 5 This is a structural block diagram of a distributed file writing device applied to input / output nodes in one embodiment;
[0085] Figure 6 This is a structural block diagram of a distributed file writing device applied to a computing node in one embodiment;
[0086] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0087] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0088] A Distributed File System (DFS) refers to a file system that manages physical storage resources that are not necessarily directly connected to local nodes, but rather connected to nodes (which can be simply understood as a computer) through a computer network. A DFS typically contains multiple compute nodes, which can be distributed file clients with data storage needs, or other servers with data storage requirements. Each compute node has data storage needs, meaning it writes data to the disk of the distributed file system for storage. Each compute node can contain multiple processes, and each process may generate data storage needs, requiring data to be written to the disk of the file storage system.
[0089] In this context, a page is the basic unit for reading and writing data to disk in a distributed file system. That is, when reading data from the disk, it is done page by page, and when writing data to the disk, it is also done page by page.
[0090] like Figure 1The diagram illustrates a distributed file system in the related technology described in this application. Each compute node can have multiple write requests. When a compute node performs a write operation on a page of a file, it needs to obtain write permission for that page from the write permission management module. After obtaining write permission, the compute node modifies the content of the page. Once the compute node confirms the completion of the modification, the write permission management module revokes the compute node's write permission for that page, allowing other compute nodes to obtain write permission for that page and perform read and write operations. Therefore, when multiple compute nodes concurrently write to the same page of the same file, the distributed file system can only execute the write requests of multiple compute nodes serially.
[0091] The main problems with distributed file writing methods in related technologies are as follows:
[0092] 1. In these write requests, for non-page aligned write requests, the compute node needs to read the old data of the entire page from the corresponding position on the disk, and then write the modified data of the entire page to the corresponding position on the disk. Therefore, the IO response time is long and the number of IO operations is large.
[0093] 2. Write permission contention leads to serialization of write operations. To ensure data integrity, distributed file systems typically deploy a write permission management module. This module manages the write permissions of each compute node for each page. For a given page, only one node can be granted write permissions for that page at any given time. Therefore, when multiple compute nodes concurrently write to the same page, only one compute node can complete its write operation before another can continue, resulting in low efficiency for distributed file writes.
[0094] Based on this, this application proposes a distributed file writing method. This method is applied to a distributed file system, which includes several computing nodes and at least one input / output node. Each page corresponds to one input / output node, and each input / output node corresponding to a page is granted write permissions for that page. The input / output node receives write requests carrying data to be written from each computing node in the distributed file system, determines the write address range for each piece of data to be written, and determines the page corresponding to each write address range. For each page, based on the address range of each piece of data to be written corresponding to the page, the data to be written to the page is aggregated to obtain the aggregated data for the page, and the aggregated data is written to the storage space corresponding to the page on disk.
[0095] The distributed file system writing method provided in this application effectively reduces the number of disk reads and writes by aggregating the data to be written from multiple write requests and then writing it to the disk. Furthermore, since each page corresponds to an input / output node, and the write permission of each page is assigned to an input / output node, when multiple computing nodes concurrently write to the same page, the write can be uniformly written to the disk by the input / output node corresponding to that page, which effectively improves the writing efficiency of multiple computing nodes concurrently writing to the same page.
[0096] The distributed file writing method provided in this application embodiment can be applied to, for example, Figure 2 The distributed 0-type file system shown includes several compute nodes, at least one input / output node, and a disk for storing data. The compute nodes and the input / output node communicate via a network, and the input / output node can perform read and write operations on the disk. The compute nodes or input / output nodes can be distributed file system clients, i.e., terminals using distributed file services to store data, or servers in other servers that need to receive write requests and write data to the disk according to the write requests.
[0097] 5. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0098] In one embodiment, such as Figure 3 As shown, a distributed file writing method is provided, applicable to, for example... Figure 2 The input / output nodes in the distributed file writing system shown include the following steps: Step 301: Receive write requests sent by each computing node, which carry data to be written.
[0099] In this context, the compute nodes are the compute nodes within the distributed file system. Each compute node sends a write request carrying data to be written, specifically a write data request for the page corresponding to its own input / output node.
[0100] Specifically, a distributed file system includes multiple compute nodes and at least one input / output node. Each page corresponds to one input / output node, and each page's corresponding input / output node is granted write permissions for that page. Therefore, if any compute node needs to modify the data on a particular page...
[0101] The data to be written will be sent to the input / output node corresponding to the page. Each input / output node will receive multiple data modification requests from computing nodes for the page corresponding to its input / output node. In other words, the input / output node receives write requests carrying data to be written from each computing node in the distributed file system.
[0102] Step 303: Determine the write address range for each piece of data to be written, and determine the page corresponding to each write address range for each piece of data to be written.
[0103] In a distributed file system, each page corresponds to an input / output node, and one input / output node can correspond to multiple pages. Therefore, multiple pieces of data to be written received by an input / output node may not be for the same page. The starting address of the write address range of the data to be written is the starting position of the data being written to disk, and the ending position is the ending position of the data being written to disk. The length of the write address range is equal to the size of the data to be written, meaning that the storage space corresponding to that address range on the disk can exactly store the data to be written corresponding to that address range. The write address range of the data to be written is within the address range of the page corresponding to that address range. For example, if the address range of page A is [0000H, 1000H], and the write address range of data X to be written is [0008H, 000CH], then the page corresponding to the address range of data X to be written is page A. If the write address range of data Y to be written is [1008H, 100CH], then the page corresponding to the address range of data Y to be written is not page A.
[0104] Specifically, for each write request carrying data to be written, the input / output nodes determine the address range targeted by the write request based on the write address information carried in the write request, that is, the address range of the data to be written carried in the write request. Based on the start address and end address of the address range of the data to be written, the nodes determine the page to which the address range of the data to be written belongs, that is, the page corresponding to the write address range of the data to be written.
[0105] Step 305: For each page, aggregate the data to be written to the page according to the address range of each data to be written to the page to obtain the aggregated data of the page.
[0106] Here, "page" refers to the page corresponding to this input / output node. For example, with 1000 pages, input / output node 1 is responsible for writing data to pages 1-200, input / output node 2 is responsible for writing data to pages 201-400, input / output node 3 is responsible for writing data to pages 401-600, input / output node 4 is responsible for writing data to pages 601-800, and input / output node 5 is responsible for writing data to pages 801-1000. If this input / output node is input / output node 2, then the pages corresponding to input / output node 2 are pages 201-400.
[0107] Specifically, for each page, the data to be written for each write request corresponding to that page has its own address range. These address ranges may or may not overlap. If the address ranges do not overlap, they may be continuous or discontinuous. For example, the address ranges for data A, B, C, and D to be written are [0000H, 0008H], [0004H, 0008H], [0008H, 000CH], and [000CH, 0010H], respectively. Among these, the address ranges for data A and data B overlap, the address ranges for data A and data C do not overlap and are continuous, and the address ranges for data A and data D do not overlap and are discontinuous.
[0108] After receiving multiple data to be written, the input / output nodes, for each page, according to the address range of each data to be written corresponding to the page, merge the two address ranges for overlapping address ranges and aggregate the corresponding data to be written. For the data to be written corresponding to the overlapping address ranges, the newer data to be written in the two overlapping address ranges overwrites the older data to be written. This process is repeated to aggregate the data to be written corresponding to the page, resulting in the aggregated data for the page.
[0109] Step 307: Write the aggregated data to the storage space corresponding to the page on the disk.
[0110] Specifically, if the address range corresponding to the aggregated data is not page-aligned (i.e., it is not the complete address range corresponding to a page), the input / output node first reads the data corresponding to the page from the disk, then modifies the data read from the disk based on the aggregated data, and finally writes the modified data to the storage space corresponding to the page on the disk. If the address range corresponding to the aggregated data is page-aligned (i.e., it is the complete address range corresponding to a page), the input / output node directly writes the aggregated data to the storage space corresponding to the page on the disk.
[0111] The distributed file system writing method provided in this application effectively reduces the number of disk reads and writes by aggregating the data to be written carried by write requests from multiple computing nodes before writing to the disk. Furthermore, since each page corresponds to an input / output node, and the write permission of each page is assigned to an input / output node, when multiple computing nodes concurrently write to the same page, the write can be uniformly written to the disk by the input / output node corresponding to that page, which effectively improves the writing efficiency of multiple computing nodes concurrently writing to the same page.
[0112] In one embodiment, after receiving a write request carrying data to be written from a compute node, the input / output node determines that the write address range of the data to be written is the address of a completed page. That is, the data to be written does not need to be aggregated with other data to be written. At this time, after the input / output node determines the page corresponding to the data to be written, it directly writes the data to be written to the storage space corresponding to that page on the disk.
[0113] In one embodiment, step 305 specifically includes:
[0114] A1. For each page, cache the data to be written to the corresponding target cache queue.
[0115] Specifically, the input / output nodes set up corresponding cache queues for each page. After receiving a write request carrying data to be written from the compute node, the input / output nodes determine the address range corresponding to the data to be written based on the address range targeted by the write request, and determine the page to which the data to be written is to be written based on the address range corresponding to the data to be written. Then, the data to be written is cached in the target cache queue corresponding to the page.
[0116] A2. If the page meets the preset first write-back conditions, the multiple data to be written are aggregated according to the address range of multiple data to be written cached in the target cache queue to obtain the aggregated data of the page.
[0117] Specifically, as the input / output nodes receive more and more data to be written, the cache queues corresponding to each page cache more and more data to be written. When the length of the target cache queue reaches the maximum length of the queue, the target cache queue cannot cache more data to be written corresponding to the page, or according to the system configuration, the input / output node is no longer responsible for writing the data to be written corresponding to the page. That is, when it is determined that the page meets the preset first write-back conditions, the multiple data to be written are aggregated according to the address range of multiple data to be written cached in the target cache queue to obtain the aggregated data of the page.
[0118] In this embodiment, the input and output nodes use a cache queue to cache the received data to be written, and the data to be written for each page is aggregated when a certain amount of data is reached.
[0119] In one instance, step 305 specifically includes:
[0120] Step B1: Add the address range of the data to be written to the address range aggregation tree according to the address range of the data to be written.
[0121] Among them, the address range aggregation tree is a data storage structure in which each node of the address range aggregation tree is used to represent the address range corresponding to a piece of data to be written.
[0122] Specifically, the input / output node sets up and maintains an address range aggregation tree for aggregating address ranges. After receiving data to be written from a compute node, the input / output node adds the address range of the data to be written to the address range aggregation tree according to the address range of the data to be written.
[0123] Step B2: Based on the address range aggregation tree, aggregate the data to be written corresponding to each address range in the address range aggregation tree according to preset rules.
[0124] Specifically, the input / output nodes traverse each address range in the address range aggregation tree. For each target address range in the address range aggregation tree, the target address range is compared with other address ranges in the tree to determine if there are any address ranges in the tree that overlap with the target address range. If an address range overlaps with the target address range, the input / output nodes merge the target address range and the overlapping address ranges in the address range aggregation tree. Based on the aggregation, they then aggregate the data to be written for both the target address range and the overlapping address ranges.
[0125] The target address range can be any address range in the address range aggregation tree.
[0126] Step B3: If the aggregated data corresponding to the page is aggregated, cache the aggregated data to the page cache queue.
[0127] It should be noted that the cache queue corresponding to each page is used to cache multiple pieces of data to be written for each page, and the page cache queue is used to cache the aggregated data corresponding to each page.
[0128] Specifically, when the input / output node maintains the address range aggregation tree, it aggregates the aggregate data corresponding to the page. This aggregate data is the data to be written after the page is completed. That is, the address range corresponding to the aggregate data is continuous and corresponds to the entire address range of the page. At this time, the aggregate data can be directly written to the storage space corresponding to the page on the disk without having to read the data corresponding to the page from the disk first, modify it, and then write it. Therefore, the input / output node can cache the aggregate data to the page cache queue.
[0129] At this point, step 307 specifically includes:
[0130] Read the aggregated data of each page in the page cache queue, and write the aggregated data of each page to the storage space corresponding to that page on the disk.
[0131] Specifically, the input and output nodes can periodically access the complete data corresponding to each page aggregated in the page cache queue, read the aggregated data of each page in the page cache queue, and write the aggregated data of each page into the storage space corresponding to the page on the disk.
[0132] In this embodiment, the input / output nodes use an address range aggregation tree to aggregate the aggregated data corresponding to each page. Only when the aggregated data corresponding to a complete page is aggregated is the aggregated data written to the disk, further reducing the number of disk accesses by the input / output nodes. Using an address range aggregation tree to organize the data buffer to be written avoids padding reads for non-page-aligned write operations, shortening I / O time. Furthermore, for multiple non-page-aligned concurrent write scenarios, page-aligned aggregated data can be aggregated, avoiding padding reads, reducing I / O counts, and improving the overall system bandwidth. In addition, after the data to be written is cached, it is written to the disk uniformly, eliminating competition for write permissions between computing nodes and fully leveraging the performance of the storage system.
[0133] In one embodiment, step B2 specifically includes:
[0134] If the address ranges corresponding to the pages to be written in the address range aggregation tree do not overlap with the address ranges corresponding to the data to be written, the address ranges of the data to be written are added to the address range aggregation tree.
[0135] Specifically, if the input / output nodes determine that there is no head overlap or tail overlap between the new address range to be written to the address range aggregation tree and any old address range in the address range aggregation tree, then the new address range is directly added to the address range aggregation tree.
[0136] In this embodiment, when the input / output node directly adds the address range corresponding to the new data to be written to the address range tree, the data to be written does not conflict with the address range of other data to be written.
[0137] In one embodiment, step B2 specifically includes:
[0138] If the address ranges corresponding to the pages to be written in the address range aggregation tree overlap with the address ranges to be written, delete the overlapping address ranges in the address range aggregation tree that overlap with the address ranges to be written, delete the data to be written corresponding to the overlapping ranges, and add the address ranges to be written to the address range aggregation tree.
[0139] Specifically, if the input / output node determines that the new address range to be written to the address range aggregation tree overlaps with the tail of an old address range in the address range aggregation tree, then the length of the old address range is modified, and the new address range is inserted. For the data to be written corresponding to the overlapping address ranges, the data to be written in the new address range overwrites the data to be written in the old address range. If the input / output node determines that the new address range to be written to the address range aggregation tree contains an old address range in the address range aggregation tree, then the old address range is deleted, and the new address range is inserted. If the input / output node determines that the new address range to be written to the address range aggregation tree overlaps with the head of an old address range in the address range aggregation tree, then the length of the old address range is modified, and the new address range is inserted. For the data to be written corresponding to the overlapping address ranges, the data to be written in the new address range overwrites the data to be written in the old address range. If the input / output nodes determine that the new address range to be written to the address range aggregation tree overlaps with the heads of multiple old address ranges in the address range aggregation tree, then the multiple old address ranges are merged, the length of the old address ranges is modified, and the new address range is inserted. For the data to be written corresponding to the overlapping address ranges, the data to be written corresponding to the new address range overwrites the data to be written corresponding to the old address range.
[0140] In this embodiment, when the input / output node adds the address range corresponding to the new data to be written to the address range tree, it dynamically adjusts each address range of the address range tree.
[0141] This application also provides a distributed file writing method, such as... Figure 4 As shown, the method is applied to, for example Figure 2 The distributed file system shown includes computing nodes, further comprising several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; the method includes:
[0142] Step 401: Determine the address range targeted by the write request and determine the page corresponding to the address range.
[0143] Specifically, several processes will run on the compute node. During the operation of each process, it may need to write data to the disk of the distributed file system, generating a write request to write data to the disk of the distributed file system. After the compute node determines that there is a write request to write data to the disk of the distributed file system, it determines the address range targeted by the write request and uses a preset address mapping strategy to map the address range to the corresponding page, thus obtaining the page corresponding to the address range targeted by the write request.
[0144] Step 403: Send the write request to the input / output node corresponding to the page, so that the input / output node can aggregate the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and write the aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0145] Specifically, compute nodes have pre-stored or are configured with a mapping between pages and input / output nodes. After determining the input / output node to which the page to which the write request belongs, the compute node obtains the network address of the input / output node corresponding to the page and sends the data to be written corresponding to the write request to the corresponding input / output node based on the obtained network address.
[0146] For example, considering 1000 pages, input / output node 1 is responsible for writing data to pages 1-200, input / output node 2 is responsible for writing data to pages 201-400, input / output node 3 is responsible for writing data to pages 401-600, input / output node 4 is responsible for writing data to pages 601-800, and input / output node 5 is responsible for writing data to pages 801-1000. The compute node determines that the address range targeted by the write request corresponds to page 300. Based on the stored mapping or configuration, it determines that input / output node 2 is responsible for writing data to page 300. Based on the network address of input / output node 2, the compute node sends the data to be written corresponding to the write request to input / output node 2.
[0147] In this embodiment, the computing nodes no longer compete with other computing nodes for write permissions on each page. Instead, they send the corresponding data to be written to the input / output nodes of the corresponding pages, reducing the time loss caused by write permission competition.
[0148] In one embodiment, step 403 specifically includes:
[0149] Step 4031: Cache the data to be written corresponding to the write request in the cache queue corresponding to the page.
[0150] Specifically, the compute nodes pre-set up corresponding cache queues for each page. After the compute node determines the page corresponding to the address range targeted by the write request, it caches the data to be written corresponding to the write request in the cache queue corresponding to the page.
[0151] Step 4033: If the page meets the preset second write conditions, aggregate the data to be written corresponding to each write request in the cache queue corresponding to the page to obtain the preliminary aggregated data of the page.
[0152] Specifically, as the compute nodes receive more and more data to be written, the cache queues corresponding to each page accumulate more and more data to be written. Either the compute node finishes its write task or a preset time is reached, the compute node determines that the page meets the preset second write conditions, reads the data to be written corresponding to each write request cached in the cache queue corresponding to the page, and aggregates the data to be written corresponding to each write request cached in the cache queue corresponding to the page to obtain the initial aggregated data of the page.
[0153] Step 4035: Send the preliminary aggregated data to the corresponding input / output nodes on the page.
[0154] Specifically, after the computing node performs preliminary aggregation on multiple data to be written on the page to obtain preliminary aggregated data, it sends the preliminary aggregated data as a single data to be written to the corresponding input / output node of the page.
[0155] In this embodiment, when a computing node has multiple write requests, the computing node aggregates the data to be written corresponding to its multiple write requests to obtain preliminary aggregated data and sends it to the input / output node. This reduces the amount of computation required for aggregation by the input / output node, reduces communication between the computing node and the input / output node, and reduces the number of transmission failures.
[0156] In one embodiment, step 4033 specifically includes:
[0157] Step A1: When the compute node determines that the cache queue corresponding to the page has reached the preset length, it aggregates the data to be written corresponding to each write request in the cache queue corresponding to the page to obtain the preliminary aggregated data of the page.
[0158] Specifically, the compute node sets a maximum length for the cache queue for each page. As the compute node receives more and more data to be written, the cache queue stores more and more data to be written. When the cache queue of a certain page reaches its maximum length, the page can no longer cache more data to be written, and the data to be written for that page will be lost. Therefore, when the compute node determines that the cache queue corresponding to the page has reached the preset length, it aggregates the data to be written corresponding to each write request in the cache queue corresponding to the page to obtain the initial aggregated data of the page.
[0159] In one embodiment, step 4033 specifically includes:
[0160] Step A2: When the preset time is reached, the computing node aggregates the data corresponding to each write request in the page queue to obtain the preliminary aggregated data of the page.
[0161] Specifically, compute nodes can periodically read and aggregate the data corresponding to each write request in the queue cache of each page to obtain the preliminary aggregated data of the page, thus avoiding the accumulation of too much data to be written in the cache queue corresponding to each page. Therefore, at the time of each period, that is, at the preset time, the compute nodes aggregate the data corresponding to each write request in the queue of the page to obtain the preliminary aggregated data of the page.
[0162] In one embodiment, step 4033 specifically includes:
[0163] Step A3: When the compute node finishes its write task, the data to be written corresponding to each write request in the cache queue of the page is aggregated to obtain the preliminary aggregated data of the page.
[0164] Specifically, if a compute node experiences a failure or configuration change, and there are no new write requests to be completed, the compute node needs to terminate its write task. At this point, the data to be written corresponding to each write request in the cache queue of the page is aggregated to obtain the preliminary aggregated data of the page.
[0165] In this embodiment, conditions for data aggregation by computing nodes under various circumstances are set to avoid the computing nodes from not being able to aggregate data for a long time or not being sent to input / output nodes for a long time.
[0166] Furthermore, this application also provides a distributed file system, which includes a plurality of computing nodes and at least one input / output node, wherein one page corresponds to one input / output node. Wherein:
[0167] The compute node is used to determine the address range targeted by the write request and the corresponding page within that address range. The write request is then sent to the corresponding input / output node for that page.
[0168] Input / output nodes receive write requests carrying data to be written from various compute nodes. They determine the write address range for each piece of data and the corresponding page for that range. For each page, based on its address range, the data to be written is aggregated to obtain the page's aggregated data. This aggregated data is then written to the storage space corresponding to the page on disk.
[0169] In one embodiment, the distributed file system further includes functional nodes. In this case, if the compute node determines that sending the data to be written has failed, it requests the address information of the input / output node corresponding to the page from the functional node. Based on the address information of the input / output node corresponding to the page returned by the functional node, the compute node sends the data to be written to the input / output node.
[0170] In one embodiment, when a functional node in a distributed file system determines that any physical input / output node has failed, it determines the logical input / output node corresponding to the physical input / output node and updates the address information of the physical input / output node corresponding to the logical input / output node.
[0171] The solution provided by the distributed file system is similar to the solution described in the above method. Therefore, the specific limitations of the distributed file system embodiments provided above can be found in the limitations of the distributed file writing method above, and will not be repeated here.
[0172] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0173] Based on the same inventive concept, this application also provides a distributed file writing device for implementing the distributed file writing method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more distributed file writing device embodiments provided below can be found in the limitations of the distributed file writing method described above, and will not be repeated here.
[0174] In one embodiment, such as Figure 5 As shown, a distributed file writing device is provided. This device is applied to the input / output nodes of a distributed file system, which further includes multiple computing nodes. The device comprises:
[0175] The receiving module 501 is used to receive write requests carrying data to be written sent by each computing node; wherein, the computing node is a computing node in the distributed file system;
[0176] The determining module 503 is used to determine the write address range of each of the data to be written, and to determine the page corresponding to each write address range of the data to be written;
[0177] The aggregation module 505 is used to aggregate the data to be written corresponding to each page according to the address range of each data to be written corresponding to the page, so as to obtain the aggregated data of the page.
[0178] The writing module 507 is used to write the aggregated data to the storage space corresponding to the page in the disk.
[0179] In one embodiment, the aggregation module 505 specifically includes:
[0180] Data caching unit A5051 (not shown in the figure) is used to cache the data to be written corresponding to each page into the target cache queue corresponding to the page for each page.
[0181] The data aggregation unit A5053 (not shown in the figure) is used to aggregate multiple data to be written according to the address range of multiple data to be written cached in the target cache queue when it is determined that the page meets the preset first write-back conditions, so as to obtain the aggregated data of the page.
[0182] In one instance, the aggregation module 505 specifically includes:
[0183] The addition unit B5051 (not shown in the figure) is used to add the address range of the data to be written to the address range aggregation tree according to the address range of the data to be written.
[0184] The interval aggregation unit B5053 (not shown in the figure) is used to aggregate the data to be written corresponding to each address interval in the address interval aggregation tree according to a preset rule.
[0185] Page caching unit B5055 (not shown in the figure) is used to cache the aggregated data corresponding to the page to the page cache queue when the aggregated data is aggregated.
[0186] At this point, the aforementioned write module 507 is specifically used for:
[0187] Read the aggregated data of each page in the page cache queue and write the aggregated data of each page into the storage space corresponding to the page on the disk.
[0188] In one embodiment, the aforementioned adding unit B5051 is specifically used for:
[0189] If it is determined that the address ranges corresponding to the pages of the data to be written in the address range aggregation tree do not overlap with the address ranges corresponding to the data to be written, the address ranges of the data to be written are added to the address range aggregation tree.
[0190] In one embodiment, the aforementioned adding unit B5051 is specifically used for:
[0191] If it is determined that the address intervals corresponding to the page to be written in the address interval aggregation tree overlap with the address intervals of the data to be written, delete the overlapping address intervals in the address interval aggregation tree that overlap with the address intervals of the data to be written, delete the data to be written corresponding to the overlapping intervals, and add the address intervals of the data to be written to the address interval aggregation tree.
[0192] In one embodiment, such as Figure 6 As shown, a distributed file writing device is provided. The device is applied to a computing node in a distributed file system. The distributed file system further includes several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node. The device includes:
[0193] The determination module 601 is used to determine the address range targeted by the write request and to determine the page corresponding to the address range;
[0194] The sending module 603 is used to send the write request to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the obtained aggregated data to the storage space corresponding to the page of the aggregated data in the disk.
[0195] In one embodiment, the sending module 603 specifically includes:
[0196] The cache unit 6031 (not shown in the figure) is used to cache the data to be written corresponding to the write request in the cache queue corresponding to the page;
[0197] Aggregation unit 6033 (not shown in the figure) is used to aggregate the data to be written corresponding to each write request in the cache queue corresponding to the page when it is determined that the page meets the preset second write conditions, so as to obtain the preliminary aggregated data of the page.
[0198] The sending unit 6035 (not shown in the figure) is used to send the preliminary aggregated data to the input / output node corresponding to the page.
[0199] In one embodiment, the aggregation unit 6033 is specifically used for:
[0200] If it is determined that the cache queue corresponding to the page has reached a preset length, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page; or
[0201] Upon reaching a preset time, the data corresponding to each write request in the page's queue is aggregated to obtain preliminary aggregated data for the page; or
[0202] When the write task is completed, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page.
[0203] Each module in the aforementioned distributed file writing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0204] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, and a communication interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a distributed file writing method.
[0205] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0206] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described above in the embodiments of the distributed file writing methods applied to input / output nodes.
[0207] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described above in the embodiments of the distributed file writing methods applied to computing nodes.
[0208] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps described above in the embodiments of the distributed file writing methods applied to input / output nodes.
[0209] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps described above in the embodiments of the distributed file writing methods applied to computing nodes.
[0210] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described above in the embodiments of distributed file writing methods applied to input / output nodes.
[0211] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described above in the embodiments of distributed file writing methods applied to computing nodes.
[0212] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0214] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0215] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A distributed file writing method, characterized in that, The method is applied to the input / output nodes of a distributed file system, where each page corresponds to one input / output node, and each input / output node corresponding to a page is granted write permissions for that page. The distributed file system further includes multiple computing nodes; the method includes: Receive write requests carrying data to be written from each computing node; wherein, the computing node is a computing node in the distributed file system; Determine the write address range for each of the data to be written, and determine the page corresponding to each write address range of the data to be written; For each page, the data to be written corresponding to each page is aggregated according to the address range of each data to be written corresponding to the page to obtain the aggregated data of the page; Write the aggregated data to the storage space corresponding to the page on the disk; The step of aggregating the data to be written corresponding to the page to obtain the aggregated data of the page includes: For overlapping address ranges, merge the overlapping address ranges and aggregate the data to be written corresponding to the overlapping address ranges.
2. The method according to claim 1, characterized in that, For each page, the aggregated data corresponding to each page is obtained by aggregating the data to be written based on the address range of each piece of data to be written for that page, including: For each page, the data to be written corresponding to the page is cached in the target cache queue corresponding to the page; If the page meets the first write-back condition, the multiple data to be written are aggregated according to the address range of the multiple data to be written cached in the target cache queue to obtain the aggregated data of the page.
3. The method according to claim 1, characterized in that, For each page, the aggregated data corresponding to each page is obtained by aggregating the data to be written based on the address range of each piece of data to be written for that page, including: Based on the address range of the data to be written, add the address range of the data to be written to the address range aggregation tree; According to the address range aggregation tree, the data to be written corresponding to each address range in the address range aggregation tree is aggregated according to a preset rule; When the aggregated data corresponding to the page is aggregated, the aggregated data is cached in the page cache queue; The step of writing the aggregated data to the storage space corresponding to the page on the disk includes: Read the aggregated data of each page in the page cache queue and write the aggregated data of each page into the storage space corresponding to the page on the disk.
4. A distributed file writing method, characterized in that, The method is applied to computing nodes in a distributed file system, which further includes several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; the input / output node corresponding to the page is granted write permissions for the page; the method includes: Determine the address range targeted by the write request, and determine the page corresponding to the address range; The write request is sent to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the aggregated data to the storage space corresponding to the page of the aggregated data in the disk. Specifically, for overlapping address ranges, the input / output nodes merge the overlapping address ranges and aggregate the data to be written corresponding to the overlapping address ranges.
5. The method according to claim 4, characterized in that, Sending the write request to the input / output node corresponding to the page includes: The data to be written corresponding to the write request is cached in the cache queue corresponding to the page; If the page meets the preset second write conditions, the data to be written corresponding to each write request in the cache queue corresponding to the page is aggregated to obtain the preliminary aggregated data of the page. The preliminary aggregated data is sent to the input / output node corresponding to the page.
6. A distributed file writing device, characterized in that, The device is applied to the input / output nodes of a distributed file system, with each page corresponding to one input / output node, and each input / output node corresponding to the page being granted write permissions for the page. The distributed file system further includes multiple computing nodes; the device includes: A receiving module is used to receive write requests carrying data to be written sent by each computing node; wherein, the computing node is a computing node in the distributed file system; The determining module is used to determine the write address range of each of the data to be written, and to determine the page corresponding to each write address range of the data to be written; The aggregation module is used to aggregate the data to be written corresponding to each page according to the address range of each data to be written corresponding to the page, so as to obtain the aggregated data of the page. The writing module is used to write the aggregated data to the storage space corresponding to the page on the disk; The aggregation module is specifically used to merge overlapping address ranges and aggregate the data to be written corresponding to the overlapping address ranges.
7. A distributed file writing device, characterized in that, The apparatus is applied to a computing node in a distributed file system, the distributed file system further including several other computing nodes and at least one input / output node, wherein one page corresponds to one input / output node; the input / output node corresponding to the page is granted write permission for the page; the apparatus includes: The determination module is used to determine the address range targeted by the write request and to determine the page corresponding to the address range; The sending module is used to send the write request to the input / output node corresponding to the page, so that the input / output node aggregates the data to be written carried by multiple write requests according to the address range targeted by the write request and the address range targeted by the write requests sent by other computing nodes, and writes the obtained aggregated data to the storage space corresponding to the page of the aggregated data in the disk. Specifically, the input / output nodes are used to merge overlapping address ranges and aggregate the data to be written corresponding to the overlapping address ranges.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3 or 4 to 5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3 or 4 to 5.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3 or 4 to 5.