Data processing system, file access method, device and related equipment

Abstract

The application discloses a data processing system, a file access method and device and related equipment, and relates to the technical field of storage. The data processing system comprises a first computing node, a second computing node, a shared memory pool and a storage node. The storage node comprises a file system. A plurality of cache pages in the shared memory pool comprise cache pages in a first memory in the first computing node and cache pages in a second memory in the second computing node. The first computing node comprises an application and a client provided by the file system. The client is used for responding to a file access request from the first application, querying an index, determining the first cache page, and accessing the first cache page to obtain a file in the file system that needs to be accessed and return the file to the application. In this way, even if the cache page in the local memory of the first computing node is not cached with the file, the file can be obtained by accessing the cache page in the memory of the other computing node, and the efficiency of file access of the computing node can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing system, file access method, apparatus and related equipment. Background Technology

[0002] Currently, computing nodes such as servers typically employ page cache mechanisms to improve data read and write performance. The data being read and written can be, for example, file-type data. Page cache, also known as page caching, is a caching mechanism used to accelerate access to persistent storage media (such as disks).

[0003] exist Figure 1 In the compute node shown, before the processor reads data from persistent storage, the data can be prefetched into the page cache in memory. This allows the processor to check if the required data exists in the page cache; if so, it can directly access the data from memory. This significantly improves the processor's data reading efficiency compared to accessing data directly from persistent storage. Furthermore, during data writing, the processor writes data to the page cache in memory, which is considered a successful write. Subsequently, the data in the page cache will be flushed to persistent storage at an appropriate time (such as when memory resources fall below a threshold), further improving the processor's data writing efficiency.

[0004] In real-world applications, data processing systems typically include multiple computing nodes. Even if each computing node in a data processing system uses a page cache mechanism for data access, the overall data access performance of the data processing system may still be at a low level. Furthermore, data processing systems are also prone to memory resource waste. Summary of the Invention

[0005] This application provides a data processing system to improve file access efficiency and memory resource utilization within the system. Furthermore, this application also provides a corresponding file access method, file access device, computing device, computer-readable storage medium, and computer program product.

[0006] Firstly, this application provides a data processing system comprising a first computing node, a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes a first application (such as a database application), first memory, and a first client provided by the file system. The second computing node includes second memory. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory. The first client is configured to receive a file access request from the first application, requesting access to a first file in the file system. The first client is also configured to respond to the file access request by querying the index corresponding to the multiple cache pages to determine the first cache page among the multiple cache pages. This index can be, for example, an XArray (extended array), which can be used to locate the cache page in the shared memory pool. The first cache page stores the first file that the first client needs to access. Furthermore, the first client is also configured to access the first cache page in the shared memory pool to obtain the required first file and return the first file to the first application to satisfy the first application's request for the first file.

[0007] Since the shared memory pool includes cached pages in the memory of the first compute node and the second compute node—meaning multiple compute nodes in the data processing system share cached pages—when a first client in the first compute node accesses the first file, even if the first file is not cached in the local memory of the first compute node, it can obtain the first file by accessing cached pages in the memory of other compute nodes. This significantly improves the efficiency of accessing the first file compared to accessing it from persistent storage. Furthermore, the granularity of data access from memory by the first client is smaller than that from persistent storage, minimizing read amplification during the first compute node's reading of the first file. This effectively improves the overall data access performance of the data processing system. Moreover, with multiple compute nodes sharing cached pages, files already cached in the memory of some compute nodes do not need to be cached again in the memory of others, avoiding redundant caching of the same files and preventing memory resource waste. In addition, the cache pages in the memory of each compute node can be accessed by multiple compute nodes, that is, clients on multiple compute nodes can read and write to the cache pages. This can avoid the cache pages from being idle for a long time because they are not accessed by the local compute node, thereby improving the utilization rate of memory resources in the data processing system.

[0008] In one possible implementation, the first compute node is configured with a local index. This local index is used to locate the cache page containing the file to be accessed among at least one cache page accessed by the first client within a past time period. The shared memory pool includes a global index, which is used to locate the cache page containing the file to be accessed among all cache pages included in the shared memory pool. Typically, the size of the global index can be larger than that of the local index. When determining the first cache page, the first client specifically queries the local index. If the first cache page is not included in at least one cache page indicated by the local index, it queries the global index to determine the first cache page from multiple cache pages. In practical applications, if the first cache page is included among the cache pages accessed by the first compute node within a past time period, the first compute node can determine the first cache page from multiple cache pages by querying the local index. Thus, the first compute node can accelerate the determination of the cache page containing the first file to be accessed using the local index, and if the local index does not indicate the first cache page, it can ensure that the required first cache page is found by querying the global index, thereby guaranteeing the reliability of file access.

[0009] In one possible implementation, when the first client accesses the first cache page, it first determines access influencing factors, including one or more of the following: the distance between the first cache page and the first compute node, the data size of the first file, and the access mode for the first file. Then, based on the access influencing factors, it determines the target access method and accesses the first cache page in the shared memory pool according to the target access method to obtain the first file. The target access method is either the processor in the first compute node accessing the file, or the copy engine in the first compute node accessing the file. Thus, the first client selects to use either the processor or the copy engine to access the first file based on its distance to the first cache page, the data size of the first file to be accessed, and one or more of the access modes for the first file, thereby ensuring the performance of the first compute node accessing the first file.

[0010] In one possible implementation, the storage nodes in the data processing system include third-party memory, and the multiple cache pages also include cache pages located in the third-party memory. In this way, clients on multiple computing nodes in the data processing system can also access the memory in the storage nodes, thereby increasing the shared memory resources and allowing a larger number of files to be cached in the shared memory pool, thus maximizing file access performance for clients within the data processing system.

[0011] In one possible implementation, the storage node includes a persistent storage medium for storing files in a file system. The data processing system also includes a cache page management device for managing indexes corresponding to multiple cache pages. This cache page management device is further configured to, according to a management policy, place files from the persistent storage medium into cache pages in a shared memory pool, migrate files between multiple cache pages, or remove some cache pages from multiple cache pages. Thus, by placing files in cache pages or migrating files between different cache pages, the cache page management device can improve the performance of client file access on the computing node. By removing some cache pages from multiple cache pages, the cache page management device can release the memory resources of those cache pages, thereby enabling flexible management of memory resources in the shared memory pool.

[0012] In one possible implementation, the cache page management device is located on the storage node. By offloading the functionality of the cache page management device to the storage node, the computational resource consumption of the compute node can be reduced, thereby helping to improve the overall computational performance of the data processing system.

[0013] In one possible implementation, the storage areas in the shared memory pool and the storage areas in the persistent storage medium are jointly addressed. In this way, the compute node can determine the actual storage location of the first file to be read with only one addressing process, thereby improving the efficiency of the compute node in reading the first file.

[0014] Secondly, this application provides a file access method applied to a data processing system. The data processing system includes a first computing node, a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes a first application, first memory, and a first client provided by the file system. The second computing node includes second memory. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory. The method includes: the first client receiving a file access request from the first application, the file access request being used to request access to a first file in the file system; the first client responding to the file access request, querying the index corresponding to the multiple cache pages, determining the first cache page among the multiple cache pages, the index being used to locate the cache page in the shared memory pool, the first cache page storing the first file that the first computing node needs to access; the first client accessing the first cache page in the shared memory pool, obtaining the first file, and returning the first file to the first application.

[0015] In one possible implementation, the first compute node is configured with a local index, which is used to locate the cache page containing the file to be accessed among at least one cache page accessed by the first client in the past time period. The shared memory pool includes a global index, which is used to locate the cache page containing the file to be accessed among all cache pages included in the shared memory pool. The first client queries the indexes corresponding to multiple cache pages to determine the first cache page among the multiple cache pages, including: the first client queries the local index; if the first cache page is not included in at least one cache page indicated by the local index, the first compute node queries the global index to determine the first cache page from the multiple cache pages.

[0016] In one possible implementation, the first client accesses a first cache page in a shared memory pool to obtain a first file, including: the first client determining access influencing factors, which include one or more of the following: the distance between the first cache page and the first computing node, the data size of the first file, and the access mode for the first file; the first computing node determining a target access method based on the access influencing factors, which is either the processor in the first computing node accessing the file or the copy engine in the first computing node accessing the file; and the first computing node accessing the first cache page in the shared memory pool to obtain the first file based on the target access method.

[0017] In one possible implementation, the storage node includes third memory, and the multiple cache pages also include cache pages located in the third memory.

[0018] In one possible implementation, the storage node includes a persistent storage medium for storing files in a file system, and the data processing system further includes a cache page management device for managing indexes corresponding to multiple cache pages; the method further includes: the cache page management device, according to a management policy, placing files in the persistent storage medium into cache pages in a shared memory pool, migrating files between multiple cache pages, or removing some cache pages from multiple cache pages.

[0019] In one possible implementation, the cache page management device is located on the storage node.

[0020] In one possible implementation, the storage regions in the shared memory pool are jointly addressed with the storage regions in the persistent storage medium.

[0021] Thirdly, this application provides a file access device applied to a first client in a first computing node of a data processing system. The data processing system further includes a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes the first application, first memory, and the first client provided by the file system. The second computing node includes second memory. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory. The file access device includes: a communication module for receiving a file access request from the first application, the file access request being used to request access to a first file in the file system; a query module for responding to the file access request, querying the index corresponding to the multiple cache pages, and determining the first cache page among the multiple cache pages, the index being used to locate the cache page in the shared memory pool, the first cache page storing the first file that the first computing node needs to access; an access module for accessing the first cache page in the shared memory pool to obtain the first file; and the communication module is further used to return the first file to the first application.

[0022] In one possible implementation, the first compute node is configured with a local index, which is used to locate the cache page containing the file to be accessed among at least one cache page accessed by the first client in the past time period. The shared memory pool includes a global index, which is used to locate the cache page containing the file to be accessed among all cache pages included in the shared memory pool. The query module is used to: query the local index; and if the first cache page is not included in at least one cache page indicated by the local index, query the global index to determine the first cache page from multiple cache pages.

[0023] In one possible implementation, the access module is configured to: determine access influencing factors, including one or more of the following: the distance between the first cache page and the first computing node, the data size of the first file, and the access mode for the first file; determine a target access method based on the access influencing factors, wherein the target access method is either the processor in the first computing node accessing the file or the copy engine in the first computing node accessing the file; and access the first cache page in the shared memory pool to obtain the first file based on the target access method.

[0024] In one possible implementation, the storage node includes third memory, and the multiple cache pages also include cache pages located in the third memory.

[0025] In one possible implementation, the storage node includes a persistent storage medium for storing files in the file system. The data processing system also includes a cache page management device for managing indexes corresponding to multiple cache pages, and for placing files in the persistent storage medium to cache pages in a shared memory pool, migrating files between multiple cache pages, or removing some cache pages from multiple cache pages according to a management policy.

[0026] In one possible implementation, the cache page management device is located on the storage node.

[0027] In one possible implementation, the storage regions in the shared memory pool are jointly addressed with the storage regions in the persistent storage medium.

[0028] Fourthly, this application provides a computing device, which includes a processor and a memory. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform a file access method as described in the second aspect or any implementation thereof. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0029] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operational steps of the file access method described in the second aspect or any implementation thereof.

[0030] Sixthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the file access method described in the second aspect or any implementation thereof.

[0031] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of an exemplary computing node;

[0033] Figure 2 A schematic diagram of the structure of an exemplary data processing system provided in this application;

[0034] Figure 3 A flowchart illustrating a file access method provided in this application;

[0035] Figure 4 A diagram illustrating the configuration of a two-level index in a data processing system;

[0036] Figure 5a This is a schematic diagram illustrating the deployment of a cache page management device on a compute node.

[0037] Figure 5b This is a schematic diagram illustrating the deployment of a cache page management device on a storage node.

[0038] Figure 5c A schematic diagram illustrating the deployment of a cache page management device independent of compute and storage nodes;

[0039] Figure 6 A schematic diagram of a file access device provided in this application;

[0040] Figure 7 This is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0041] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0042] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0043] See Figure 2 This is a schematic diagram of the structure of an exemplary data processing system 20. Figure 2 As shown, the data processing system 20 includes multiple computing nodes and at least one storage node. Different computing nodes can be connected via a bus, and computing nodes and storage nodes can also be connected via a bus. For ease of understanding and explanation, Figure 2 The following example illustrates the use of compute node 100, compute node 200, and storage node 300.

[0044] The computing node in the data processing system 20 refers to a node with data computing capabilities, which can be implemented by computing devices including processors and memory, such as servers.

[0045] For example, a processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0046] The memory can be random access memory (RAM) or read-only memory (ROM). RAM can be, for example, dynamic random access memory (DRAM), static random access memory (SRAM), or storage class memory (SCM). ROM can be, for example, programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). Additionally, the memory can be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM); or, high bandwidth memory (HBM); or, solid state disk (SSD). Furthermore, the shared memory 300 can also have a power-saving function, meaning that the data stored in the memory will not be lost when power is lost and then restored.

[0047] Each compute node can run one or more applications. Figure 2 The example provided illustrates running two applications per compute node. For instance, the applications on a compute node can be business applications, such as database applications, and are not limited to this. Furthermore, each compute node can also run a client provided by the file system 303, such as... Figure 2 As shown, compute node 100 can also run client 105, and compute node 200 can also run client 205. The file system 303 runs on storage node 300, and can be used to store data as files in storage node 300 and manage the files in storage node 300. Within the compute nodes, the processor can support the execution of various applications and the clients provided by file system 303.

[0048] In addition to the file system 303, the storage node 300 may also include a persistent storage medium 301 and memory 302. For example, the persistent storage medium 301 refers to a storage medium with the ability to persistently store data, such as a disk, optical disk, or other types of storage medium. Specifically, the files in the file system 303 may be stored in the persistent storage medium 301.

[0049] The bus connecting different compute nodes, compute nodes and storage nodes can be any type of UB bus in the Unified Bus Clan (UBC), or it can be a ComputeExpress Link (CXL) bus, an Ultra Accelerator Link (UALink) bus, or a Peripheral Component Interconnect Express (PCIe) bus, or other types of buses, without limitation.

[0050] exist Figure 2 In the data processing system 20 shown, each computing node can use a page cache mechanism for data reading and writing. Taking computing node 100 as an example, when client 105 reads a file from storage node 300 (for example, based on a request from application 103), it can first check if the file to be read is stored in the cache page in memory. If it is, client 105 can directly read the file from the cache page. If not, client 105 needs to read the file from the persistent storage medium 301 in storage node 300. Since the performance of client 105 accessing memory is higher than the performance of client 105 accessing the persistent storage medium 301 in storage node 300, computing node 100 can improve file access efficiency by using the page cache mechanism to read files.

[0051] However, if the clients in compute node 100 and compute node 200 can only access the cache pages in their respective memories, the efficiency improvement of accessing files by a single compute node is poor, and the data processing system 20 is prone to problems such as wasted memory resources and low utilization of memory resources.

[0052] For example, since the storage space of memory 102 in compute node 100 is limited, that is, the number of files that memory 102 can store is limited, the files that compute node 100 wants to access have a high probability of not being cached in memory 102. As a result, compute node 102 will frequently read the files it needs from persistent storage medium 301 of storage node 300, resulting in a poor overall improvement in the efficiency of compute node 100 accessing files.

[0053] Furthermore, when compute nodes 100 and 200 frequently access the same file, since each compute node can only access its own cached pages in memory, the file is cached not only in memory 102 of compute node 100 for fast access, but also in memory 202 of compute node 200 for fast access. Thus, the same file is stored in separate copies in the memories of different compute nodes, resulting in wasted memory resources.

[0054] Furthermore, when the frequency of file reading by compute node 100 is higher than that of file reading by compute node 200, there may be a shortage of memory resources in compute node 100. At the same time, the memory 202 in compute node 200 may be idle for a long time because it is not frequently accessed. This will lead to a large difference in memory load among different compute nodes and a low overall utilization rate of memory resources in data processing system 20.

[0055] Based on this, in the data processing system 20 provided in this application, when the page cache mechanism is adopted, multiple computing nodes can share memory with each other, so that each computing node can access the cache pages in the memory of multiple computing nodes.

[0056] In specific implementation, such as Figure 2 As shown, the data processing system 20 may include a shared memory pool 400, which includes cache pages in memory 102 and cache pages in memory 202. Each compute node can access any cache page in the shared memory pool 400. The multiple cache pages in the shared memory pool 400 can be configured with corresponding indexes, such as the data structure XArray (extended array). These indexes can be used to locate cache pages in the shared memory pool 400, specifically to locate the cache page containing the file that the compute node needs to access from among the multiple cache pages included in the shared memory pool 400.

[0057] Taking an application 103 in compute node 100 requesting file access as an example, application 103 can send a file access request to client 105 provided by file system 303. This file access request is used to request access to a file in file system 303. Client 105 responds to the file access request by querying the indices corresponding to multiple cache pages in shared memory pool 400 to determine whether shared memory pool 400 includes a cache page storing the file that application 103 needs to read. This cache page could be a cache page in memory 102 or a cache page in memory 202. Assuming that a cache page storing the file exists in shared memory pool 400, client 105 can access this cache page to read the required file and return the read file to application 103.

[0058] Because multiple computing nodes in the data processing system 20 share cache pages in memory, when client 105 accesses a file, even if the cache page in the local memory 102 of computing node 100 does not cache the file required by application 103, it can still obtain the file by accessing the cache page in the memory 202 of computing node 200. This effectively improves the efficiency of client 105 reading files compared to reading files from the persistent storage medium 301 of storage node 300, thus improving the efficiency of application 103 obtaining files. Furthermore, the granularity of data accessed by client 105 from memory is smaller than the granularity of data accessed from persistent storage medium 301, which minimizes read amplification during data reading. Therefore, the overall data access performance of the data processing system 20 is effectively improved.

[0059] Similarly, when client 205 in compute node 200 reads a file, it can also use a similar method to read the required file from the cache pages in shared memory pool 400.

[0060] Furthermore, when compute nodes 100 and 200 share cache pages in memory, files already cached in the memory of some compute nodes do not need to be cached again in the memory of others. This avoids the data processing system repeatedly caching the same files using memory resources, thus preventing memory resource waste. For example, when compute nodes 100 and 200 frequently access the same file, assuming the file is cached in memory 102 on compute node 100, not only can compute node 100 quickly read the file from memory 102, but compute node 200 can also quickly read the file by accessing memory 102. This ensures the performance of file reading by compute nodes 100 and 200 while preventing the same file from being stored multiple times in the shared memory pool 400, thereby mitigating or preventing memory resource waste in the data processing system 20.

[0061] Furthermore, the cache pages in the memory of each compute node can be accessed by multiple compute nodes, meaning multiple compute nodes can read and write to the cache page. This minimizes the possibility of the cache page remaining idle for extended periods due to inaccessibility by the local compute node, thus improving the utilization of memory resources in the data processing system. For example, if compute node 100 reads and writes files more frequently than compute node 200, compute node 200 might not access memory 202 for an extended period. However, if compute node 100's local memory 102 is insufficient, it can utilize the memory 202 in compute node 200 for file reading and writing. This minimizes the possibility of compute node 200's memory 202 remaining idle for extended periods, thereby improving the utilization of memory resources in the data processing system 20.

[0062] Furthermore, Figure 2 In the data processing system 20 shown, the shared memory pool 400 may also include memory 302 in the storage node 300. That is, each computing node can access not only local memory and memory in other computing nodes, but also memory 302 in the storage node 300, thereby increasing the shared memory resources.

[0063] Figure 2The data processing system 20 shown may further include a cache page management device 401. This device 401 can manage the indexes corresponding to multiple cache pages in the shared memory pool 400. For example, the cache page management device 401 can create the index for multiple cache pages in the shared memory pool 400, or update the index when the cache pages in the shared memory pool 400 change. Additionally, the cache page management device 401 can also manage the cache pages in the shared memory pool 400, such as updating file-type data in the cache pages or removing some cache pages from the shared memory pool 400.

[0064] The cache page management device 401 can be implemented in software or hardware. When implemented in software, the cache page management device 401 can be an application running on hardware, such as a process running on a compute node / storage node. When implemented in hardware, the cache page management device 401 can be a processor, or a computing device including a processor, etc., and there is no limitation in this regard.

[0065] As some examples, in practical applications... Figure 2 The data processing system 20 shown can be a cluster, such as a cluster used for distributed training of artificial intelligence (AI) models; or, the data processing system 20 can be a super pod in a cluster, in which case the computing node can specifically be a processor; or, the data processing system 20 can be a rack server, which refers to a server architecture designed to be compatible with standard racks, in which case the computing node can specifically be a server. In this embodiment, the specific product form of the data processing system 20 in actual application scenarios is not limited.

[0066] It is worth noting that the above Figure 2 The data processing system 20 shown is for illustrative purposes only and is not intended to be limiting. For example, in other possible implementations, all or some of the computing nodes in the data processing system 20 may be configured with a copy engine, which may be, for example, a direct memory access (DMA) engine or other type of engine, and may be used to access files in the memory of other computing nodes or storage nodes across nodes.

[0067] For example, in other possible implementations, the data processing system 20 may include three or more computing nodes, and some computing nodes may share all or part of their memory space with other computing nodes, or they may not share their memory space. Different computing nodes and computing nodes and storage nodes may also be connected via a network.

[0068] For example, in other possible implementations, the storage node 300 in the data processing system 20 may only include the persistent storage medium 301, that is, the storage node 300 does not include the memory 302.

[0069] For ease of understanding, embodiments of the file access method provided in this application will be described below with reference to the accompanying drawings.

[0070] See Figure 3 , Figure 3 This is a flowchart illustrating a file access method provided in an embodiment of this application. This method can be applied to... Figure 2 The data processing system 20 shown may also be applied to other suitable data processing systems. For ease of explanation, this embodiment uses a system applied to... Figure 2 The data processing system 20 shown is illustrated by way of example, with application 103 in computing node 100 requesting to read a file.

[0071] in, Figure 3 The file access method shown may specifically include the following steps.

[0072] S301: The cache page management device 401 configures corresponding indexes for multiple cache pages in the shared memory pool 400.

[0073] In this embodiment, when memory 102 in computing node 100 and memory 202 in computing node 200 are shared, cache page management device 401 can configure corresponding indexes for cache pages in memory that are shared and accessed by multiple computing nodes, so that the cache pages can be located using the indexes in the future.

[0074] For example, the index configured in the cache page management device 401 can be, for instance, a data structure called XArray. XArray is an array structure used to locate the cache page containing the file to be read from multiple cache pages. It can include the address information of each cache page and information indicating the file cached by that cache page (such as the offset of the data in the file within the file). When the cache page does not store a file, XArray can only record the address information of each cache page, with the rest of the information empty. When the cache page stores a file, XArray can simultaneously include the address information of each cache page, as well as the file identifier and information corresponding to the cache page storing the file.

[0075] In practical applications, XArray can also include cache page status information and access records. The status information can be used to indicate whether the cache page is valid, modified, or locked, etc., and the access records can include the number of times the cache page has been accessed and the timestamp of the access. In this embodiment, the information content included in XArray is not limited.

[0076] In one possible implementation, during the creation of the shared memory pool 400, the cache page management device 401 obtains the memory space that each compute node 100 is allowed to share. For example, a compute node 100 (such as a processor 101 in compute node 100) can send a notification message to the cache page management device 401, which may include the storage space in memory 102 that is allowed to be accessed by compute node 200. The shared memory space in each compute node can be all the storage space in that compute node's memory or a portion of that storage space; there is no limitation on this. For ease of understanding and description, this embodiment uses the example of all the storage space in memory 102 and memory 202 being shared.

[0077] Then, the cache page management device 401 can create multiple cache pages for the shared memory space in compute node 100 and compute node 200. The memory space size corresponding to each cache page can be a preset size, such as 4KB (kilobytes). Furthermore, the cache page management device 401 can further create an XArray for the multiple cache pages to manage them.

[0078] Thus, in the data processing system 20, a shared memory pool 400 can be constructed based on the memory 102 in the computing node 100 and the memory 202 in the computing node 200.

[0079] In a further possible implementation, when storage node 300 includes memory 302, some or all of the storage space in memory 302 of storage node 300 can also be shared by compute node 100 and compute node 200. Then, cache page management device 401 can also acquire the shared memory space in storage node 300 and create cache pages and corresponding indexes for this portion of memory space. Thus, the constructed shared memory pool 400 can include memory 102, memory 202, and memory 302, thereby increasing the storage space in the shared memory pool 400.

[0080] S302: Application 103 in compute node 100 sends a file access request to client 105, which is used to request access to the first file in file system 303.

[0081] S303: Client 105 responds to the file access request by querying the indexes corresponding to multiple cache pages and determining the first cache page among the multiple cache pages. This first cache page stores the first file that client 105 needs to access.

[0082] In this embodiment, for ease of distinction and description, the file that the client 105 needs to access is referred to as the first file. If the first file is cached in the shared memory pool 400, the cache page where the first file is located is referred to as the first cache page.

[0083] In real-world applications, application 103 on compute node 100 may need to read files during operation to execute subsequent business logic. In this case, application 103 can read the required files through client 105.

[0084] In a specific implementation, application 103 can generate a file access request, which may include indication information of the first file in file system 303 to be read, such as the identifier of the first file, to request the reading of the first file. Then, application 103 can send the file access request to client 105 so that client 105 can read the corresponding first file based on the file access request.

[0085] Since the efficiency of client 105 in reading files from memory is generally higher than that in reading files from persistent storage medium 301 in storage node 300, client 105 can access the memory space in shared memory pool 400 when responding to file access requests, so that if the first file is stored in shared memory pool 400, the first file can be read directly from memory, thereby improving the efficiency of client 105 in reading files.

[0086] Specifically, client 105 can respond to the file access request by querying the index pre-configured by cache page management device 401 to determine whether shared memory pool 400 caches the first file that client 105 needs to read, and if shared memory pool 400 caches the first file, determine the first cache page that stores the first file from multiple cache pages.

[0087] Taking the index as a data structure XArray as an example, client 105 can use the indication information of the first file as the key, traverse the XArray starting from the root node to determine the node containing the key, and determine the cache page pointed to by the value (e.g., a pointer) in that node as the first cache page containing the first file. Furthermore, it determines the address information of the first cache page, which can be the virtual address or physical address of the first cache page, without limitation. The first cache page can be one cache page or multiple cache pages.

[0088] It is understandable that if client 105 traverses all nodes in XArray according to the key and finds no node that matches the key, this indicates that the shared memory pool 400 does not cache the first file. Therefore, client 105 can obtain the first file by accessing the persistent storage medium 301 in storage node 300.

[0089] In real-world applications, when the size of the memory space in the shared memory pool 400 is large, the number of cache pages included in the shared memory pool 400 is also large. Consequently, the amount of data in the index used to manage these multiple cache pages is also large. In this case, the time taken for client 105 to query this index to determine the first cache page may be quite long.

[0090] Based on this, in a further possible implementation, the cache page management device 401 can be configured with a two-level index, namely a global index and a local index, such as... Figure 4As shown. The global index is used to locate the cache page containing the file that a client (which could be client 105 in compute node 100 and / or client 205 in compute node 200) needs to access, among all the cache pages included in the shared memory pool 400. The local index is used to locate the cache page containing the file that the client currently needs to access, among at least one cache page accessed by the client in a compute node within a past time period. Each compute node can be configured with a separate local index. In practical applications, since the cache pages accessed by client 105 and client 205 within a past time period may differ, the local index configured in compute node 100 may differ from the local index configured in compute node 200. Typically, since clients within a single compute node may only access a portion of the memory space in the shared memory pool 400 within a past time period, the amount of data in the local index configured in that compute node will be less than the amount of data in the global index.

[0091] Accordingly, when client 105 determines that the first cache page of the first file is stored in shared memory pool 400, it can first query... Figure 4 The local index in the computing node 100 shown can be queried based on the indication information of the first file. Furthermore, if the cached pages accessed by the client 105 within a past time period include a first cached page storing the first file, the client 105 can determine the first cached page by accessing a local index with a smaller data volume. This improves the efficiency of the client 105 in determining the first cached page, thereby improving the overall efficiency of the client 105 in reading the file.

[0092] If the first cached page is not among the cached pages accessed by client 105 within the past time period, meaning client 105 cannot find the first cached page using the local index, then client 105 can continue querying. Figure 4 The global index shown is used to locate the first cache page in the shared memory pool 400 where the first file is stored. Furthermore, after client 105 determines the first cache page by querying the global index, cache page management device 401 can update the local index in compute node 100 based on the first cache page. Thus, when client 105 subsequently reads the first file from the first cache page, it can retrieve the first cache page using the updated local index, thereby improving the efficiency of subsequent file reading by client 105.

[0093] S304: Client 105 accesses the first cache page in shared memory pool 400 to obtain the first file to be read.

[0094] S305: Client 105 returns the first file to application 103.

[0095] After determining the first cache page, the client 105 can access the first cache page according to the address information of the first cache page to read the first file stored in the first cache page, and return the read first file to the application 103 to meet the application 103's requirement for the first file.

[0096] In this embodiment, the following three implementation examples for reading the first file are provided.

[0097] In the first implementation example, the storage space in memory indicated by the first cache page could be a segment of storage space on memory 102 in compute node 100, or it could be a segment of storage space on memory 202 in compute node 200. Furthermore, when memory 302 in storage node 300 can be shared and accessed by multiple compute nodes, the storage space in memory indicated by the first cache page could also be a segment of storage space on memory 302 in storage node 300. In this case, client 105 can access the file in the cache page using different methods for cache pages located at different positions.

[0098] In specific implementation, after determining the first cache page, client 105 can calculate the distance between the first cache page and compute node 100 based on the address information of the first cache page. This distance can be the physical distance between the memory device where the first cache page is located and compute node 100, etc. For example, when the distance between the first cache page and compute node 100 is greater than 0, it indicates that the first cache page is located in memory 202 of compute node 200 or memory 302 of storage node 300; when the distance between the first cache page and compute node 100 is equal to 0, it indicates that the first cache page is located in memory 102 of compute node 100. In practical applications, the distance between the first cache page and compute node 100 can also be defined in other ways, and there is no limitation on this.

[0099] Then, client 105 can determine, based on the distance between the first cache page and compute node 100, whether to use processor 101 to read the first file from the first cache page or to use the copy engine in compute node 100. Figure 2 (Not shown in the image) Reads the first file from the first cache page. For example, the copy engine could be, for instance, a DMA engine, etc.

[0100] For example, when the distance between the first cache page and compute node 100 is greater than 0, client 105 can use the copy engine to access the first cache page across nodes to read the first file stored in the first cache page. When the distance between the first cache page and compute node 100 is equal to 0, client 105 can use processor 101 to access the storage area in memory 102 indicated by the first cache page through a load method to read the first file in that storage area. Specifically, processor 101 can generate a load instruction and access memory 102 according to the memory address included in the load instruction (that is, the memory address of the first cache page in memory 102) to read the first file.

[0101] Since the efficiency of using the copy engine to read the first file in the memory of other nodes is higher than that of the processor 101 when the distance between the first cache page and the compute node 100 is large, the client 105 can improve the efficiency of reading the first file by using different methods to read the first file of the cache page for different locations.

[0102] In the second implementation example, client 105 can access the cache page in different ways for files of different data sizes.

[0103] When the amount of data in the first file that client 105 needs to read is small (for example, it can be determined based on IO size), client 105 can use processor 101 to access the first cache page to read the first file stored in the first cache page. When the amount of data in the first file that client 105 needs to read is large, client 105 can use the copy engine to access the first cache page to read the first file.

[0104] Since copy engines are typically suitable for reading large amounts of data, while processor 101 is suitable for reading small amounts of data, when the amount of data in the first file to be read is large, the efficiency of using the copy engine to read the first file is usually higher than that of using processor 101. Conversely, when the amount of data in the first file to be read is small, the efficiency of using processor 101 to read the first file is usually higher than that of using the copy engine. Therefore, depending on the size of the first file, client 105 can adopt an appropriate method to read the first file, thereby improving the efficiency of client 105 in reading the first file.

[0105] In the third implementation example, client 105 can access cached pages in different ways depending on the different data access patterns.

[0106] The data access mode can include, for example, sequential access or random access. Sequential access refers to accessing data in the storage area in a specific order. Random access refers to being able to access data at any location in the storage area randomly, without needing to follow a sequence. Therefore, when client 105 uses random access mode to read the first file, client 105 can use processor 101 to access the first cache page to read the first file stored in the first cache page. Conversely, when client 105 uses sequential access mode to read the first file, client 105 can use the copy engine to access the first cache page to read the first file.

[0107] Since the copy engine is typically suited for sequential data access, while the processor 101 is suited for random data access, when sequential access is used to read the first file, the efficiency of using the copy engine is usually higher than that of using the processor 101. Conversely, when random access is used, the efficiency of using the processor 101 is usually higher than that of using the copy engine. Therefore, the client 105 uses different methods to read the first file for different data access modes, thereby improving the efficiency of the client 105 in reading the first file.

[0108] It is understood that the above three implementation examples are merely illustrative and not intended to limit the scope. In other embodiments, client 105 may also determine the method for reading the first file based on other implementation methods. For example, client 105 may combine the above multiple implementation examples. For instance, when sequentially accessing a first file with a large amount of data, client 105 may determine to use a copy engine to read the first file; when randomly accessing a first file with a small amount of data, client 105 may determine to use processor 101 to read the first file.

[0109] In practical applications, when determining the access method to read the first file, client 105 can first determine the access influencing factors. These factors may be one or more of the following: the distance between the first cache page and the computing node 100, the amount of data in the first file to be accessed, and the access mode for the first file; or other factors that can affect how client 105 accesses the first file. Then, based on these factors, client 105 determines the target access method for the first file. This target access method may be the method of accessing the file using processor 101, the method of accessing the file using a copy engine, or other types of data access methods. Therefore, client 105 can access the first cache page in the shared memory pool 400 to obtain the first file according to the target access method.

[0110] Furthermore, after reading the first file, client 105 can generate a response message for the file access request, which may carry the first file. Client 105 can then send this response message to application 103. Correspondingly, application 103 can parse the received first file to obtain the required file.

[0111] In addition, the following points are provided for explanation regarding this embodiment.

[0112] (1) In this embodiment, the example given is that the shared memory pool 400 contains a first cache page with the first file. In actual application scenarios, the first file that the client 105 needs to read may not be cached in the shared memory pool 400. In this case, after querying the index, the client 105 determines that the first file that the client 105 needs to read does not exist in the shared memory pool 400. At this time, the client 105 can access the persistent storage medium 301 in the storage node 300 through the bus to read the first file from the persistent storage medium 301. For example, the first file in the persistent storage medium 301 can be directly read into the buffer of the application 103.

[0113] Furthermore, after client 105 obtains the first file from persistent storage medium 301, cache page management device 401 can load the first file from persistent storage medium 301 into cache pages in shared memory pool 400 and update the indexes corresponding to the multiple cache pages included in shared memory pool 400. When cache page management device 401 is configured with two levels of indexes, it can update not only the global index but also the local index configured in compute node 100. Thus, when client 105 accesses the first file again subsequently, it can read the first file from shared memory pool 400 by querying the local index, without needing to read the first file from persistent storage medium 301 again. In addition, when client 205 in compute node 200 also needs to access the first file, even if the local index configured in compute node 200 does not find the cache page storing the first file, it can still read the first file from the shared memory pool 400 by querying the global index, without having to read the first file from the persistent storage medium 301, thereby improving the efficiency of client 205 reading the first file.

[0114] Furthermore, the storage areas (memory areas) in the shared memory pool 400 can be jointly addressed with the storage areas in the persistent storage medium 301 of the storage node 300. Specifically, the logical addresses of the storage areas in the shared memory pool 400 and the logical addresses of the storage areas in the persistent storage medium 301 are jointly addressed. Thus, when the client 105 performs addressing based on the logical address of the first file to be read, if the first file is located in the shared memory pool 400, the client 105 can determine the physical address of the first file in the shared memory pool 400 through addressing; if the first file is located in the persistent storage medium 301, the client 105 can determine the physical address of the first file in the persistent storage medium 301 through addressing. In this way, the client 105 only needs to perform one addressing process to determine the actual storage location of the first file to be read, thereby improving the efficiency of the client 105 in reading the first file.

[0115] (2) In this embodiment, the first cache page accessed by the client 105 can be a single cache page. Alternatively, the first cache page can be multiple cache pages, meaning that different parts of the data in the first file can reside in different cache pages. In this case, the client 105 can access the data in these multiple cache pages sequentially to obtain all the data included in the first file.

[0116] Furthermore, when client 105 needs to read files from multiple cache pages simultaneously, such as when application 103 and application 104 request to read files at the same time, client 105 can perform parallel queries on the index corresponding to shared memory pool 400 to determine multiple cache pages. Different cache pages are used to store data from different files, so client 105 can perform the process of reading files from multiple cache pages in parallel.

[0117] (3) In this embodiment, the example of client 105 reading a file from shared memory pool 400 is used for illustration. In actual application, client 105 can write files to shared memory pool 400 in the same manner as described above. Files successfully written to shared memory pool 400 can be subsequently flushed to persistent storage medium 301 at an appropriate time for persistent storage and are managed by file system 303. The following is a combination of... Figure 3 The following describes the process of client 105 writing the file sent by application 103 to shared memory pool 400.

[0118] S306: Application 103 sends a file write request to client 105, which is used to request the writing of a second file to file system 303.

[0119] S307: Client 105 responds to the file write request by querying the indexes corresponding to multiple cache pages and determining the second cache page among the multiple cache pages. This second cache page is used to cache the second file that client 105 needs to write.

[0120] S308: Client 105 writes the second file to the second cache page.

[0121] The second cache page may be a segment of storage space on memory 102 in compute node 100, or it may be a segment of storage space on memory 202 in compute node 200. Furthermore, when memory 302 in storage node 300 can be shared and accessed by multiple compute nodes, the storage space in memory indicated by the first cache page may also be a segment of storage space on memory 302 in storage node 300.

[0122] Similar to the above implementation of reading the first file from the cache page, the client 105 can also determine the target access method when writing the second file to the second cache page based on access influencing factors. The target access method can be the method of writing the second file using the processor 101, or the method of writing the second file using the copy engine.

[0123] For example, after determining the second cache page, client 105 can calculate the distance between the second cache page and compute node 100 based on the address information of the second cache page. This distance could be the physical distance between the memory device where the second cache page resides and compute node 100. Furthermore, when the distance between the second cache page and compute node 100 indicates that the second cache page is located in memory 102, client 105 can use processor 101 to access the second cache page in memory 102 indicated by the second cache page to write a second file to that second cache page. For example, processor 101 can generate a store instruction and write the second file to memory 102 based on the memory address included in the load instruction (i.e., the memory address of the second cache page in memory 102). When the distance between the second cache page and compute node 100 indicates that the second cache page is located in memory 202 or memory 302, client 105 can use the copy engine to access the second cache page across nodes to write a second file to that second cache page. Thus, for cached pages at different locations, client 105 uses different methods to write the second file to the cached page, thereby improving the file writing efficiency of client 105.

[0124] For example, client 105 can determine the amount of data in the second file to be written. When the amount of data in the second file to be written is small (e.g., determined based on I / O size), client 105 can use processor 101 to access the second cache page to write the second file to the second cache page. When the amount of data in the second file to be written is large, client 105 can use the copy engine to access the second cache page to write the second file to the second cache page. In this way, depending on the size of the second file, client 105 uses an appropriate method to write the second file to the second cache page, thereby improving the file writing efficiency of client 105.

[0125] For example, client 105 can determine the access mode of the memory region during file writing. When client 105 accesses the second cache page in random access mode, it can utilize processor 101 to access the second cache page and write the second file into it. When client 105 accesses the second cache page in sequential access mode, it can utilize the copy engine to access the second cache page and write the second file into it. Thus, client 105 uses different methods to write the second file for different access modes, thereby improving its file writing efficiency.

[0126] In practical applications, such as Figure 3 As shown, after successfully writing the second file to the second cache page, client 105 can also send a write response to application 103 to notify application 103 that the file has been written successfully.

[0127] Furthermore, such as Figure 3 As shown, after client 105 writes the second file to shared memory pool 400, cache page management device 401 can also update the index corresponding to shared memory pool 400, such as updating the global index and the local index corresponding to compute node 100. In this way, when client 105 subsequently requests to read the second file from shared memory pool 400, it can query the index to determine the cache page storing the second file and read the required second file from that cache page, thereby improving the efficiency of client 105 accessing the second file.

[0128] In this embodiment, the cache page management device 401 can not only manage the indexes corresponding to multiple cache pages in the shared memory pool 400, but also manage the cached files and cache pages in the shared memory pool 400.

[0129] As a first implementation example, the cache page management device 401 can place files in the persistent storage medium 301 into cache pages in the shared memory pool 400 according to the file management policy.

[0130] In practice, the cache page management device 401 can obtain information such as input / output (IO) traces and / or access patterns of each computing node. For example, each computing node can record the generated IO and / or the access patterns used during access to the shared memory pool 400 and persistent storage medium 301, and provide this information to the cache page management device 401. Then, based on the IO traces and / or access patterns and a pre-configured management policy, the cache page management device 401 can determine whether to place files in the file system 303 from the persistent storage medium 301 to cache pages in the shared memory pool 400.

[0131] For example, the cache page management device 401 can determine whether a file in the persistent storage medium 301 is frequently accessed by the compute nodes based on IO traces. If so (e.g., the number of times the file is accessed exceeds a threshold within a unit of time), the cache page management device 401 can place the file from the persistent storage medium 301 into a cache page in the shared memory pool 400. If not, the cache page management device 401 will not place the file into the shared memory pool 400.

[0132] For example, the cache page management device 401 can determine, based on IO traces and access patterns, that a client (such as client 105 or client 205) is reading a file in persistent storage medium 301 in a sequential access manner, and that the amount of data in the file being read is greater than a threshold. In this case, the cache page management device 401 can place the file into a cache page in the shared memory pool 400.

[0133] For example, the cache page management device 401 can determine, based on the access mode, when a client (such as client 105 or client 205) continuously reads different parts of the file in the persistent storage medium 301 in a sequential access manner, the cache page management device 401 can predict the remaining data in the file that the client needs to read in the next access to the persistent storage medium 301 (such as data in the subsequent position stored sequentially with the already read data), and prefetch the remaining data into the cache pages in the shared memory pool 400 to improve the overall efficiency of the client reading the file.

[0134] Furthermore, when placing a file from persistent storage medium 301 into shared memory pool 400, cache page management device 401 can also determine which cache page to place the file in. For example, cache page management device 401 can count the number of times each cache page in shared memory pool 400 is accessed, so that the compute node can place the file in a cache page with fewer accesses, thereby achieving balanced usage across multiple cache pages. Alternatively, cache page management device 401 can place the file in a cache page that is relatively close to the compute node to which the client accessing the file belongs, further improving the efficiency of the client reading files from shared memory pool 400.

[0135] After placing the file in the persistent storage medium 301 into the cache page in the shared memory pool 400, the cache page management device 401 can also update the index corresponding to the shared memory pool 400 so that subsequent clients can read the file from the shared memory pool 400 by querying the updated index.

[0136] As a second implementation example, the cache page management device 401 can migrate a file between multiple cache pages in the shared memory pool 400 according to the file management policy.

[0137] In practice, the cache page management device 401 can obtain information such as the IO traces of each computing node, and determine whether to perform file migration between multiple cache pages based on the IO traces and a pre-configured management strategy.

[0138] For example, the cache page management device 401 can determine, based on I / O traces, that the compute node 100 frequently accesses files in cache page 1 of the shared memory pool 400. Furthermore, when the distance between cache page 1 and the compute node 100 is large (e.g., cache page 1 is a cache page on memory 202 of the compute node 200 or a cache page on memory 302 of the storage node 300), the cache page management device 401 can migrate the file from cache page 1 to cache page 2 located on memory 102 of the compute node 100. In this way, the client 105 can subsequently read the file by accessing cache page 2. Moreover, the distance between the compute node 100 and cache page 2 is less than the distance between the compute node 100 and cache page 1, which improves the efficiency of the client 105 accessing the file after migrating the file from cache page 2 to cache page 1.

[0139] In practical applications, the cache page management device 401 can also determine whether to migrate files between multiple cache pages based on access patterns, or by combining access patterns and IO traces, from dimensions such as the balance of cache page usage and the amount of data in the files accessed by the computing nodes, without imposing any restrictions on this.

[0140] After migrating files from different cache pages in the shared memory pool 400, the cache page management device 401 can also update the index corresponding to the shared memory pool 400 so that subsequent compute nodes can read the file required by the client from the correct cache page in the shared memory pool 400 by querying the index.

[0141] As a third implementation example, the cache page management device 401 can eviction some cache pages in the shared memory pool 400 according to the management policy for cache pages.

[0142] In practice, the cache page management device 401 can obtain information such as the IO traces of each computing node, and determine whether to remove some cache pages in the shared memory pool 400 based on the IO traces and a pre-configured management policy.

[0143] For example, the cache page management device 401 can determine, based on I / O traces, that some cache pages in the shared memory pool 400 have not been accessed for a long time or have been accessed less than a preset number of times within a unit time period. The files on these cache pages can be considered cold data, and the cache page management device 401 can then remove these cache pages from the shared memory pool 400, thereby releasing the memory resources corresponding to them. The released memory resources can then be used by the compute node for other purposes, such as allocating them to newly created processes or threads within the compute node.

[0144] In practical applications, the cache page management device 401 can also determine whether to remove a cache page based on the access pattern, or by combining the access pattern and IO traces, from dimensions such as the balance of cache page usage and the amount of data in the file accessed by the client, without imposing any limitations on this.

[0145] After removing some cached pages from the shared memory pool 400, the cache page management device 401 can also update the index corresponding to the shared memory pool 400 to prevent errors from occurring during index queries by clients (such as client 105 or client 205).

[0146] It should be noted that the above-described implementation of the cache page management device 401 for managing cached files and cache pages in the shared memory pool 400 is only an example and is not intended to limit the scope. In other embodiments, the cache page management device 401 may also use other methods to manage cached files and cache pages.

[0147] In this way, the cache page management device 401 can manage cache pages globally (that is, in memory shared by multiple nodes), which can improve the management efficiency and quality of memory resources in the shared memory pool 400, and help improve the overall data read and write performance of the data processing system 20.

[0148] In practical applications, the cache page management device 401 can be deployed in Figure 2 At any point in the data processing system 20 shown below, in conjunction with... Figures 5a to 5c This paper introduces three implementation examples of the deployment location of the cache page management device 401 in the data processing system 20.

[0149] In the first implementation example, the cache page management device 401 can be deployed on a compute node. For example... Figure 5a As shown, both compute node 100 and compute node 200 can be equipped with a cache page management device 401. The cache page management device 401 deployed in each compute node is responsible for managing the local index corresponding to that compute node. Furthermore, as... Figure 5a As shown, the cache page management device 401 in each compute node, or the cache page management device 401 in one of the compute nodes, can also be responsible for managing the global index corresponding to the shared memory pool 400, as well as managing the files and cache pages in the shared memory pool 400, such as placing files in the persistent storage medium 301 to the shared memory pool 400, or migrating files on different cache pages in the shared memory pool 400, or removing some cache pages in the shared memory pool 400.

[0150] In the second implementation example, the cache page management device 401 can be deployed on the storage node. For example... Figure 5b As shown, a cache page management device 401 can be deployed in storage node 300. Furthermore, the cache page management device 401 is responsible not only for managing the local indexes in each compute node, but also for managing the global index corresponding to the shared memory pool 400. Simultaneously, the cache page management device 401 can also manage files and cache pages in the shared memory pool 400.

[0151] When the cache page management device 401 is implemented in software, its functions can be executed using the computing engine in the storage node 300. By offloading the functions of the cache page management device 401 to the storage node 300, the computational resource consumption of the computing node can be reduced, thereby helping to improve the overall computational performance of the data processing system 20.

[0152] In the third implementation example, the cache page management device 401 can be deployed independently of individual compute nodes and storage nodes. For example... Figure 5cAs shown, the cache page management device 401 can be deployed outside of compute node 100, compute node 200, and storage node 300. In this case, the cache page management device 401 can be implemented using a separate computing device, or the cache page management device 401 can be deployed as software on that separate computing device. Furthermore, the computing device can be connected to each compute node and storage node via a bus, or via a network; there is no limitation on this.

[0153] In practical applications, the cache page management device 401 can also be implemented using other applicable deployment methods, and there is no limitation on this. For example, the cache page management device 401 may include a functional module 1 for managing local indexes and a functional module 2 for managing global indexes and cache pages and files in the shared memory pool 400. Thus, each compute node can be deployed with a separate functional module 1 to manage the local indexes in each compute node. The storage node 300 (or an independent computing device) can be deployed with functional module 2 to manage the global indexes and cache pages and files in the shared memory pool 400.

[0154] In this embodiment, multiple computing nodes share cache pages in memory. This allows a client on any computing node to access a file even if the file is not cached in its local memory cache page. It can obtain the file by accessing the cache pages in the memory of other computing nodes (or storage nodes). Compared to the way computing nodes access files from persistent storage medium 301, this not only effectively improves the efficiency of client file access but also minimizes read amplification during the process of the client reading files from persistent storage medium 301, thereby improving the overall data access performance of the data processing system 20.

[0155] Meanwhile, only one copy of the same file needs to be stored in the shared memory pool 400, eliminating the need to store it separately in the memory of each computing node. This avoids wasting memory resources. Furthermore, the memory resources in the shared memory pool 400 are greater than the local memory resources of each computing node. This not only makes it easier to meet the memory resource usage needs of individual computing nodes but also prevents some memory resources from remaining idle for extended periods due to lack of access from a computing node, thereby improving the utilization rate of memory resources in the data processing system 20.

[0156] Furthermore, the page cache mechanism in the data processing system 20 is transparent to applications on the compute nodes, which eliminates the need for technicians to modify applications on each compute node to adapt to the shared memory pool 400. Moreover, even if some cached pages in the shared memory pool 400 fail or malfunction, applications can continue to use the remaining cached pages without affecting their normal operation.

[0157] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0158] The above combination Figures 1 to 5c The file access method provided in the embodiments of this application will be introduced. Next, the structure of the file access device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0159] See Figure 6 The diagram shows a schematic of a file access device. Figure 6 The file access device 600 shown is applied to a first computing node in a computing data processing system. The data processing system also includes a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes a first application, first memory, and a first client provided by the file system. The second computing node includes second memory. In actual application, the second computing node may also include a second application and a second client provided by the file system. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory.

[0160] like Figure 6 As shown, the file access device 600 includes:

[0161] Communication module 601 is used to receive a file access request from a first application, the file access request being used to request access to a first file in the file system;

[0162] The query module 602 is used to respond to file access requests, query the index corresponding to multiple cache pages, and determine the first cache page among multiple cache pages. The index is used to locate the cache page in the shared memory pool. The first cache page stores the first file that the first computing node needs to access.

[0163] Access module 603 is used to access the first cache page in the shared memory pool to obtain the first file;

[0164] The communication module 601 is also used to return the first file to the first application.

[0165] In one possible implementation, the first computing node is configured with a local index, which is used to locate the cache page containing the file to be accessed in at least one cache page accessed by the first client in the past time period. The shared memory pool includes a global index, which is used to locate the cache page containing the file to be accessed in all cache pages included in the shared memory pool.

[0166] Query module 602 is used for:

[0167] Query the local index;

[0168] If the first cached page is not included in at least one cached page indicated by the local index, the global index is queried to determine the first cached page from multiple cached pages.

[0169] In one possible implementation, the access module 603 is used for:

[0170] Determine the access influencing factors, which include one or more of the following: the distance between the first cache page and the first computing node, the data size of the first file, and the access mode for the first file.

[0171] Based on the factors affecting access, the target access method is determined. The target access method is either the processor in the first computing node accessing the file or the copy engine in the first computing node accessing the file.

[0172] Based on the target access method, access the first cache page in the shared memory pool to obtain the first file.

[0173] In one possible implementation, the storage node includes third memory, and the multiple cache pages also include cache pages located in the third memory.

[0174] In one possible implementation, the storage node includes a persistent storage medium for storing files in the file system. The data processing system also includes a cache page management device for managing indexes corresponding to multiple cache pages, and for placing files in the persistent storage medium to cache pages in a shared memory pool, migrating files between multiple cache pages, or removing some cache pages from multiple cache pages according to a management policy.

[0175] In one possible implementation, the cache page management device is located on the storage node.

[0176] In one possible implementation, the storage regions in the shared memory pool are jointly addressed with the storage regions in the persistent storage medium.

[0177] because Figure 6 The file access device 600 shown corresponds to the above. Figure 3The client 105 in the illustrated embodiment, therefore Figure 6 For a detailed description of the implementation of the file access device 600 and its technical effects, please refer to the above. Figure 3 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.

[0178] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application. Figure 7 As shown, the computing device 700 includes a processor 701, a memory 702, a communication interface 703, and a bus 704. The processor 701, memory 702, and communication interface 703 communicate via the bus 704. The bus 704 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by only one thick line, but this does not indicate that there is only one bus or one type of bus. Communication interface 703 is used for communication with external devices.

[0179] It should be understood that in the embodiments of this application, processor 701 may be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. General-purpose processors may be microprocessors or any conventional processors, etc.

[0180] The memory 702 may include read-only memory and random access memory, and provides instructions and data to the processor 701. The memory 702 may also include non-volatile random access memory.

[0181] The memory 702 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0182] The memory 702 stores executable code, and the processor 701 executes the executable code to perform the aforementioned actions. Figure 3 The method executed by client 105 in the illustrated embodiment.

[0183] It should be understood that the computing device 700 according to the embodiments of this application may correspond to the client 105 in the embodiments of this application, and may correspond to the execution of the application according to the embodiments of this application. Figure 3 The method executed by client 105 in the illustrated method, and the above and other operations and / or functions implemented by computing device 700, are respectively for the purpose of implementing... Figure 3 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0184] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned file access method.

[0185] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0186] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0187] The computer program product can be a software installation package. If any of the aforementioned file access methods are required, the computer program product can be downloaded and executed on a computing device.

[0188] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0189] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0190] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0191] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing system, characterized in that, The data processing system includes a first computing node, a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes a first application, a first memory, and a first client provided by the file system. The second computing node includes a second memory. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory. The first client is configured to receive a file access request from the first application, the file access request being used to request access to a first file in the file system; In response to the file access request, the index corresponding to the plurality of cache pages is queried to determine the first cache page among the plurality of cache pages. The index is used to locate the cache page in the shared memory pool. The first cache page stores the first file that the first client needs to access. The first client is also configured to access the first cache page in the shared memory pool, obtain the first file, and return the first file to the first application.

2. The data processing system according to claim 1, characterized in that, The first computing node is configured with a local index, which is used to locate the cache page where the file to be accessed is located in at least one cache page accessed by the first client in the past time period. The shared memory pool includes a global index, which is used to locate the cache page where the file to be accessed is located in all cache pages included in the shared memory pool. The first client is used for: Query the local index; If the first cache page is not included in at least one cache page indicated by the local index, the global index is queried to determine the first cache page from the plurality of cache pages.

3. The data processing system according to claim 1 or 2, characterized in that, The first client is used for: Determine access influencing factors, which include one or more of the following: the distance between the first cache page and the first computing node, the data volume of the first file, and the access mode for the first file; Based on the access influencing factors, the target access method is determined, wherein the target access method is either the processor in the first computing node accessing the file or the copy engine in the first computing node accessing the file; According to the target access method, the first cache page in the shared memory pool is accessed to obtain the first file.

4. The data processing system according to any one of claims 1 to 3, characterized in that, The storage node includes a third memory, and the plurality of cache pages also include cache pages located in the third memory.

5. The data processing system according to any one of claims 1 to 4, characterized in that, The storage node includes the persistent storage medium, which is used to store files in the file system. The data processing system also includes a cache page management device, which is used to manage the indexes corresponding to the multiple cache pages. The cache page management device is further configured to, according to a management policy, place files in the persistent storage medium into cache pages in the shared memory pool, or migrate files between multiple cache pages, or remove some cache pages from the multiple cache pages.

6. The data processing system according to claim 5, characterized in that, The cache page management device is located on the storage node.

7. The data processing system according to claim 5 or 6, characterized in that, The storage areas in the shared memory pool are jointly addressed with the storage areas in the persistent storage medium.

8. A file access method, characterized in that, The method is applied to a data processing system, which includes a first computing node, a second computing node, a shared memory pool, and a storage node. The storage node includes a file system. The first computing node includes a first application, a first memory, and a first client provided by the file system. The second computing node includes a second memory. The shared memory pool includes multiple cache pages, including cache pages located in the first memory and cache pages located in the second memory. The method includes: The first client receives a file access request from the first application, the file access request being used to request access to a first file in the file system; The first client responds to the file access request by querying the index corresponding to the plurality of cache pages and determining the first cache page among the plurality of cache pages. The index is used to locate the cache page in the shared memory pool. The first cache page stores the first file that the first client needs to access. The first client accesses the first cache page in the shared memory pool to obtain the first file; The first client returns the first file to the first application.

9. The method according to claim 8, characterized in that, The first computing node is configured with a local index, which is used to locate the cache page where the file to be accessed is located in at least one cache page accessed by the first client in the past time period. The shared memory pool includes a global index, which is used to locate the cache page where the file to be accessed is located in all cache pages included in the shared memory pool. The first client queries the indexes corresponding to the plurality of cached pages to determine the first cached page among the plurality of cached pages, including: The first client queries the local index; If the first cached page is not included in at least one cached page indicated by the local index, the first client queries the global index to determine the first cached page from the plurality of cached pages.

10. The method according to claim 8 or 9, characterized in that, The first client accesses the first cache page in the shared memory pool to obtain the first file, including: The first client determines access influencing factors, which include one or more of the following: the distance between the first cache page and the first computing node, the data volume of the first file, and the access mode for the first file. The first client determines the target access method based on the access influencing factors, wherein the target access method is either the processor in the first computing node accessing the file or the copy engine in the first computing node accessing the file; The first client accesses the first cache page in the shared memory pool according to the target access method to obtain the first file.

11. The method according to any one of claims 8 to 10, characterized in that, The storage node includes a third memory, and the plurality of cache pages also include cache pages located in the third memory.

12. The method according to any one of claims 8 to 11, characterized in that, The storage node includes the persistent storage medium, which is used to store files in the file system. The data processing system also includes a cache page management device, which is used to manage the indexes corresponding to the multiple cache pages. The method further includes: According to the management policy, the cache page management device places files in the persistent storage medium into cache pages in the shared memory pool, or migrates files between multiple cache pages, or removes some cache pages from the multiple cache pages.

13. The method according to claim 12, characterized in that, The cache page management device is located on the storage node.

14. The method according to claim 12 or 13, characterized in that, The storage areas in the shared memory pool are jointly addressed with the storage areas in the persistent storage medium.

Images