File caching method and device, electronic equipment, storage medium and program product
By building a shared memory pool in big data and AI applications and dynamically managing the Page Cache, the problems of poor caching performance and data redundancy on computing nodes are solved, and the dynamic expansion of the memory pool and the sharing of cache files across multiple nodes are realized.
Patent Information
- Application Number
- CN202410571286.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-09
- Publication Date
- 2025-11-11
AI Technical Summary
In data-intensive applications such as big data and AI, the Page Cache of each computing node occupies local memory, resulting in poor caching performance and data redundancy.
By constructing a global shared memory pool and aggregating the Page Cache, memory nodes in the memory pool allocate or release memory according to the needs of the compute nodes. The Page Cache of the cached files is determined using index tables and index trees, thereby realizing dynamic expansion of the memory pool and eliminating data redundancy between multiple nodes.
It enables multiple computing nodes to share cache files, dynamically expand memory capacity, solves the problem of poor caching performance, and eliminates data redundancy between multiple nodes.
Smart Images
Figure CN120929432A_ABST
Abstract
Description
Technical Field
[0001] This application relates to computer caching technology, and more particularly to a file caching method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] Page cache is used to cache page data of files. It reads or caches file data from the hard drive into memory, so that when the file needs to be read again, it can be read directly from memory. Page cache effectively improves the efficiency of reading files from the hard drive.
[0003] Currently, in data-intensive application scenarios such as big data and AI, each dataset file is shared by multiple computing nodes. Each computing node has its own Page Cache, which occupies limited local memory for storage. When the available memory reaches a threshold, a Swap mechanism is triggered, causing the cached file data to be swapped from memory back to disk.
[0004] Therefore, current file caching methods suffer from poor caching performance. Summary of the Invention
[0005] This application provides a file caching method, apparatus, electronic device, storage medium, and program product to achieve the effects of dynamic expansion of the memory pool and sharing of cached data among multiple nodes.
[0006] In a first aspect, embodiments of this application provide a file caching method, including:
[0007] The request to retrieve the cached file includes the target name of the first cached file and the offset of the first cached file.
[0008] Based on the target name of the first cached file, the target base address is obtained from the index table. The target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree.
[0009] Based on the offset of the first cache file and the target base address, the target index tree and the address of the second cache file in the target index tree in the memory pool are determined, wherein the second cache file represents the block content of the first cache file;
[0010] Based on the address of the second cache file in the memory pool in the target index tree, perform read and write operations on the second cache file.
[0011] In one possible implementation, the method further includes:
[0012] When the target base address is not obtained from the index table, determine the target memory node in the memory pool and the target base address of the target memory node;
[0013] Based on the target name of the first cached file, a target index tree corresponding to the target name is obtained, wherein the target index tree and the target base address have a corresponding relationship;
[0014] Based on the target base address and the target name of the first cache file, the initial index table in the target node is updated to obtain the index table;
[0015] Based on the index table, the steps are as follows: obtaining the target base address from the index table based on the target name of the first cache file, and then performing read and write operations on the second cache file based on the address of the second cache file in the memory pool from the target index tree.
[0016] In one possible implementation, when the target base address is not obtained from the index table, determining the target memory node in the memory pool and the target base address of the target memory node includes:
[0017] When the target base address is not found in the index table, the name of the first cache file is hashed to obtain a hash value;
[0018] Based on the hash value, the list of memory nodes, and the encoding information of the memory nodes in the list of memory nodes, we obtain the target memory node, the target number information of the target memory node, and the target base address of the target memory node. The list of memory nodes is a list of memory nodes in the memory pool.
[0019] In one possible implementation, the target memory node, the target number information of the target memory node, and the target base address of the target memory node are obtained based on the hash value, the memory node list, and the encoding information of the memory nodes in the memory node list, including:
[0020] Based on the encoding information of the memory nodes in the memory node list, determine the initial hash value corresponding to the encoding information of the memory nodes in the memory node list;
[0021] Determine the target hash value based on the hash value;
[0022] Determine the target memory node based on the target hash value;
[0023] Based on the list of memory nodes and the target memory node, obtain the target number information of the target memory node and the target base address of the target memory node.
[0024] In one possible implementation, determining the target index tree and the address of the second cache file in the memory pool within the target index tree, based on the offset of the first cache file and the target base address, includes:
[0025] Determine the target index tree based on the target base address;
[0026] Based on the offset of the first cache file, determine the initial address corresponding to the target pointer in the target index tree;
[0027] If the cache file corresponding to the initial address is the second cache file, then the initial address is determined to be the address of the second cache file in the memory pool.
[0028] In one possible implementation, after determining the initial address corresponding to the target pointer in the target index tree based on the offset of the first cache file, the method further includes:
[0029] If the cache file corresponding to the initial address is not the second cache file, then obtain the second cache file from the file system;
[0030] Based on the second cache file, the cache file corresponding to the initial address is updated and replaced. After updating and replacing the cache file corresponding to the initial address, the initial address is determined to be the address of the second cache file in the memory pool.
[0031] In one possible implementation, before performing read / write operations on the second cache file based on the address of the second cache file in the memory pool according to the target index tree, the method further includes:
[0032] Send a connection request to the memory management node in the global file cache system, and the memory management node will establish a connection with the target node after receiving the connection request;
[0033] After the connection between the memory management node and the target node is established, the metadata information of the memory nodes in the memory pool is obtained from the memory management node;
[0034] Based on the metadata information of the memory nodes in the memory pool, establish an RDMA connection with the memory nodes.
[0035] In one possible implementation, the address of the second cache file in the memory pool is obtained based on the offset of the first cache file and the base address of the index tree corresponding to the name of the first cache file, including:
[0036] If the address of the second cache file in the memory pool is not found in the index tree, then the data of the second cache file in the file system is retrieved.
[0037] Based on the offset of the first cache file, determine the pointer corresponding to the second cache file in the index tree;
[0038] Based on the pointer corresponding to the second cache file in the index tree, the data of the second cache file is stored at the address corresponding to the pointer in the index tree, so that the address of the second cache file in the memory pool exists in the index tree.
[0039] Secondly, embodiments of this application provide a file caching device, comprising:
[0040] The retrieval module is used to retrieve requests for cached files. The requests for retrieving cached files include the target name of the first cached file and the offset of the first cached file.
[0041] The first determining module is used to obtain the target base address from the index table according to the target name of the first cached file. The target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree.
[0042] The second determining module is used to determine the target index tree and the address of the second cache file in the memory pool based on the offset of the first cache file and the target base address, wherein the second cache file represents the block content of the first cache file;
[0043] The read / write module is used to perform read and write operations on the second cache file based on the address of the second cache file in the memory pool in the target index tree.
[0044] In one possible implementation, it also includes:
[0045] When the target base address is not obtained from the index table, determine the target memory node in the memory pool and the target base address of the target memory node;
[0046] Based on the target name of the first cached file, a target index tree corresponding to the target name is obtained, wherein the target index tree and the target base address have a corresponding relationship;
[0047] Based on the target base address and the target name of the first cache file, the initial index table in the target node is updated to obtain the index table;
[0048] Based on the index table, the steps are as follows: obtaining the target base address from the index table based on the target name of the first cache file, and then performing read and write operations on the second cache file based on the address of the second cache file in the memory pool from the target index tree.
[0049] In one possible implementation, when the target base address is not obtained from the index table, determining the target memory node in the memory pool and the target base address of the target memory node includes:
[0050] When the target base address is not found in the index table, the name of the first cache file is hashed to obtain a hash value;
[0051] Based on the hash value, the list of memory nodes, and the encoding information of the memory nodes in the list of memory nodes, we obtain the target memory node, the target number information of the target memory node, and the target base address of the target memory node. The list of memory nodes is a list of memory nodes in the memory pool.
[0052] In one possible implementation, the target memory node, the target number information of the target memory node, and the target base address of the target memory node are obtained based on the hash value, the memory node list, and the encoding information of the memory nodes in the memory node list, including:
[0053] Based on the encoding information of the memory nodes in the memory node list, determine the initial hash value corresponding to the encoding information of the memory nodes in the memory node list;
[0054] Based on the hash value, the target hash value;
[0055] Determine the target memory node based on the target hash value;
[0056] Based on the list of memory nodes and the target memory node, obtain the target number information of the target memory node and the target base address of the target memory node.
[0057] In one possible implementation, determining the target index tree and the address of the second cache file in the memory pool within the target index tree, based on the offset of the first cache file and the target base address, includes:
[0058] Determine the target index tree based on the target base address;
[0059] Based on the offset of the first cache file, determine the initial address corresponding to the target pointer in the target index tree;
[0060] If the cache file corresponding to the initial address is the second cache file, then the initial address is determined to be the address of the second cache file in the memory pool.
[0061] In one possible implementation, after determining the initial address corresponding to the target pointer in the target index tree based on the offset of the first cache file, the apparatus further includes:
[0062] If the cache file corresponding to the initial address is not the second cache file, then obtain the second cache file from the file system;
[0063] Based on the second cache file, the cache file corresponding to the initial address is updated and replaced. After updating and replacing the cache file corresponding to the initial address, the initial address is determined to be the address of the second cache file in the memory pool.
[0064] In one possible implementation, before performing read / write operations on the second cache file based on the address of the second cache file in the memory pool according to the target index tree, the method further includes:
[0065] Send a connection request to the memory management node in the global file cache system, and the memory management node will establish a connection with the target node after receiving the connection request;
[0066] After the connection between the memory management node and the target node is established, the metadata information of the memory nodes in the memory pool is obtained from the memory management node;
[0067] Based on the metadata information of the memory nodes in the memory pool, establish an RDMA connection with the memory nodes.
[0068] In one possible implementation, the address of the second cache file in the memory pool is obtained based on the offset of the first cache file and the base address of the index tree corresponding to the name of the first cache file, including:
[0069] If the address of the second cache file in the memory pool is not found in the index tree, then the data of the second cache file in the file system is retrieved.
[0070] Based on the offset of the first cache file, determine the pointer corresponding to the second cache file in the index tree;
[0071] Based on the pointer corresponding to the second cache file in the index tree, the data of the second cache file is stored at the address corresponding to the pointer in the index tree, so that the address of the second cache file in the memory pool exists in the index tree.
[0072] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0073] The memory stores the instructions that the computer executes;
[0074] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0075] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0076] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0077] The file caching method, apparatus, electronic device, storage medium, and program product provided in this application aggregates the global Page Cache into a shared memory pool. The memory nodes in the memory pool can allocate or release memory according to the needs of the computing nodes. This allows the computing nodes to determine the Page Cache corresponding to the cached file based on the index table and index tree according to their needs for obtaining the cached file. As a result, the memory pool can be dynamically expanded, enabling multiple computing nodes to share the cached file, eliminating data redundancy between multiple nodes, and thus solving the problem of poor caching performance. Attached Figure Description
[0078] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0079] Figure 1 A schematic diagram illustrating the scenario of the shared memory pool provided in this application;
[0080] Figure 2 A flowchart illustrating a file caching method provided in this application;
[0081] Figure 3 A schematic diagram of the index tree structure provided in this application;
[0082] Figure 4 A flowchart illustrating another file caching method provided in this application;
[0083] Figure 5 A schematic diagram of the file caching device provided in this application;
[0084] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.
[0085] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0086] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0087] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
[0088] Existing technologies use page cache to store data from disk in memory, reducing disk I / O operations and thus improving the performance of retrieving cached files. However, the most direct drawback of page cache using local memory is its limited capacity. Page cache requires local memory space, and data-intensive applications such as big data and AI also consume a large amount of memory, reducing the application's available memory and potentially triggering swapping (exchanging memory to disk for later retrieval). Another drawback of current page cache is redundancy. Each compute node has its own page cache. For files, especially in AI training scenarios, dataset files are shared across multiple compute nodes. Therefore, different compute nodes are likely to cache the same files.
[0089] To address two shortcomings of current Page Cache, this application utilizes a shared memory pool to construct a global Page Cache. Compute nodes are separated from the shared memory pool, allowing multiple compute nodes to share the same pool. The file caching method provided in this application aggregates the global Page Cache into the shared memory pool. Memory nodes in the pool can allocate or release memory according to the needs of compute nodes. This allows compute nodes to determine the corresponding Page Cache for a cached file based on the index table and index tree, thereby achieving dynamic expansion of the memory pool. This enables multiple compute nodes to share cached files, eliminating data redundancy between nodes and thus resolving the issue of poor caching performance.
[0090] First, let me explain the terms used in this application:
[0091] A node is a term referring to a class of devices, which can be a host (PC), a server, or a switch, router, firewall, etc., that makes up a transmission network. A computing node is a basic unit based on a distributed computing architecture. It has independent computing resources and can execute various computing tasks. A computing node can be a physical server, a virtual machine, or a container, and it works in collaboration with other nodes to complete tasks. Each computing node has its own processor, memory, and storage devices to achieve efficient computing.
[0092] Page Cache is a mechanism for caching disk data in memory. It caches data in units of pages, thereby improving the speed of accessing disk data.
[0093] RDMA (Remote Direct Memory Access) was developed to address the latency issues in server-side data processing during network transmission. RDMA directly transfers data over the network to the computer's memory, quickly moving data from one system to a remote system's memory without affecting the operating system. RDMA provides an I / O-based channel that allows an application to directly read and write to remote virtual memory via an RDMA device.
[0094] Figure 1 The schematic diagram of the shared memory pool provided in this application is applied to a global file caching system, such as... Figure 1 As shown, the memory pool mainly consists of two types of nodes: memory nodes and management nodes. Memory nodes are responsible for providing memory, and the index tree is stored in the memory nodes. Management nodes, also known as memory management nodes, are responsible for maintaining the memory pool's metadata. Management nodes and memory nodes can be deployed independently or in a mixed deployment. The memory pool's metadata includes a list of memory nodes and the memory usage of each memory node. Compute nodes establish connections with the memory pool via an RDMA network and can perform memory allocation, read, and write operations on the memory nodes by calling APIs.
[0095] Based on the above scenarios, it can be seen that in the existing technology, multiple computing nodes have their own Page Cache, resulting in data redundancy among multiple nodes and causing poor caching performance.
[0096] This application utilizes a shared memory pool to construct a global Page Cache, separating compute nodes from the shared memory pool, allowing multiple compute nodes to share the same shared memory pool. The file caching method provided in this application aggregates the global Page Cache into the shared memory pool. Memory nodes in the pool can allocate or release memory according to the needs of compute nodes. This allows compute nodes to determine the corresponding Page Cache for a cached file based on the index table and index tree, thereby achieving dynamic expansion of the memory pool, enabling multiple compute nodes to share cached files, and eliminating data redundancy between multiple nodes.
[0097] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0098] Figure 2 This application provides a flowchart illustrating a file caching method applied to a target node in a global file caching system, such as... Figure 1 As shown, the method includes:
[0099] S201, Request to retrieve cached files. The request to retrieve cached files includes the target name of the first cached file and the offset of the first cached file.
[0100] The target node can refer to a node capable of performing various tasks. In this application, the target node can be a computing node.
[0101] A request can refer to the request information initiated by the application to the target node. In this application, a request to obtain a cache file can be a request initiated by the application to the target node to obtain a cache file. The information attached to the request includes the name of the cache file, the offset of the cache file, and the length of the cache file.
[0102] A cache file can refer to a file cached in the memory pool of a global file caching system. In this application, the first cache file can be the cache file corresponding to the cache file name in the request information, and the cache file can be a page cache file.
[0103] The target name can refer to the name of the cached file in the request information.
[0104] A page cache file consists of multiple pages, and the offset can refer to the position of the page in the file. In this application, the offset of the first cache file can refer to the position of the page that the application needs to obtain in the cache file.
[0105] S202. Based on the target name of the first cached file, obtain the target base address from the index table, where the target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree.
[0106] The index table can refer to a table that establishes a correspondence between column values and record rows based on one or more columns of a table in a certain order. In this application, the index table can be a hash table. In the hash table, the key represents a unique identifier, and the value represents the data associated with the key. In this application, the key value is the cache file name, and the value value is the base address of the index tree corresponding to the cache file name.
[0107] An index tree can represent a tree-like index relationship data structure. In this application, the cache file is divided into multiple page cache files. The index tree can be a radix tree. The child nodes of the radix tree contain count, height, and slots fields. count represents the number of child nodes in the radix tree, height represents the height of the child node from the bottom layer to the current layer in the radix tree, and the last layer of the child node contains the address information corresponding to the page cache file. The address is located by the pointer slots. By summing up the page cache files corresponding to each child node, a complete cache file can be obtained.
[0108] The target index tree can refer to the index tree corresponding to the base address of the index tree in the index table for the cached file name in the request information, and the target base address can refer to the base address of the index tree corresponding to the cached file name in the index table for the request information.
[0109] In one possible implementation, the method further includes:
[0110] When the target base address is not obtained from the index table, determine the target memory node in the memory pool and the target base address of the target memory node;
[0111] Based on the target name of the first cached file, a target index tree corresponding to the target name is obtained, wherein the target index tree and the target base address have a corresponding relationship;
[0112] Based on the target base address and the target name of the first cache file, the initial index table in the target node is updated to obtain the index table;
[0113] Based on the index table, the steps are as follows: obtaining the target base address from the index table based on the target name of the first cache file, and then performing read and write operations on the second cache file based on the address of the second cache file in the memory pool from the target index tree.
[0114] The memory pool can represent memory management using a pool approach. In this application, the memory pool includes a memory management node and multiple memory nodes. The memory management node maintains a list of memory nodes and information about each memory node in the memory node table. The memory pool establishes a connection with the target node through an API (Application Programming Interface).
[0115] A memory node can refer to a node used to manage memory, while a target memory node can refer to a memory node used to store the target index tree.
[0116] There is a correspondence between the target index tree and the target base address; the base address of the target index tree can be used as the target base address.
[0117] The initial index table can refer to an index table that does not contain target base address information for the target index tree. Updating the initial index table means adding the target name corresponding to the key value and the target base address corresponding to the key value of the first cache file to the initial index table to obtain the index table.
[0118] In one possible implementation, when the target base address is not obtained from the index table, determining the target memory node in the memory pool and the target base address of the target memory node includes:
[0119] When the target base address is not found in the index table, the name of the first cache file is hashed to obtain a hash value;
[0120] Based on the hash value, the list of memory nodes, and the encoding information of the memory nodes in the list of memory nodes, we obtain the target memory node, the target number information of the target memory node, and the target base address of the target memory node. The list of memory nodes is a list of memory nodes in the memory pool.
[0121] The memory node list can refer to the list of memory nodes in the memory pool, which includes the numbering information of the memory nodes.
[0122] The target number information of the target memory node can refer to the IP (Internet Protocol Address) address of the target memory node. In this application, assuming the hash space is 0 to 2^32-1, each segment of the IP address is converted into an 8-bit binary number and finally concatenated into a 32-bit hash value, which is the encoding information of the memory node. Each memory node's IP corresponds to an encoding information, forming a hash ring.
[0123] Hash processing can refer to performing a hash operation on the name of the first cache file. In this application, the MD5 hash algorithm can be used to perform the operation and obtain the MD5 (MD5Message-DigestAlgorithm) value. The lower 32 bits of the MD5 value are taken as the hash value corresponding to the MD5 value. At this time, the hash value corresponding to the MD5 value can fall on the hash ring. The memory node corresponding to the encoding information of the memory node closest to the hash value corresponding to the MD5 value is the target memory node.
[0124] The target base address of the target memory node can refer to the target base address of the target index tree of the target memory node.
[0125] In one possible implementation, the target memory node, the target number information of the target memory node, and the target base address of the target memory node are obtained based on the hash value, the memory node list, and the encoding information of the memory nodes in the memory node list, including:
[0126] Based on the encoding information of the memory nodes in the memory node list, determine the initial hash value corresponding to the encoding information of the memory nodes in the memory node list;
[0127] Determine the target hash value based on the hash value;
[0128] Determine the target memory node based on the target hash value;
[0129] Based on the list of memory nodes and the target memory node, obtain the target number information of the target memory node and the target base address of the target memory node.
[0130] The initial hash value can refer to the hash value corresponding to the encoded information of the memory nodes in the memory node list.
[0131] The target hash value can refer to the target hash value that corresponds to the initial hash value of the encoded information of the memory node in the memory node list. In this application, the target hash value can refer to the hash value corresponding to the encoded information of the memory node that is closest to the MD5 value.
[0132] S203. Based on the offset of the first cache file and the target base address, determine the target index tree and the address of the second cache file in the memory pool within the target index tree, wherein the second cache file represents the block content of the first cache file.
[0133] In one possible implementation, determining the target index tree and the address of the second cache file in the memory pool within the target index tree, based on the offset of the first cache file and the target base address, includes:
[0134] Determine the target index tree based on the target base address;
[0135] Based on the offset of the first cache file, determine the initial address corresponding to the target pointer in the target index tree;
[0136] If the cache file corresponding to the initial address is the second cache file, then the initial address is determined to be the address of the second cache file in the memory pool.
[0137] In this application, the first cache file is divided into multiple page cache files, and the second cache file can refer to the block content of the first cache file. An example of the target index tree is shown below. Figure 2 As shown, offset represents the offset of the page in the file. Offset is divided into bits, with 6 bits as a unit, which is used as the index of slots to locate the next child node, until the bottom layer is found. For example, when the target base address root of the target index tree has been determined by the index table, the offset of the first cache file is 64, and the offset is represented in binary as 000001000000. Starting from root, the pointer of the child node is determined as slots[0] based on bits 0-5 of the offset. slots[0] points to the first node of the second level of the index tree; the pointer of the child node is determined as slots[1] based on bits 6-11 of the offset. At this time, since height=1, it indicates that the query is over. The position pointed to by the pointer slots[1] of the bottom child node is the address of the second cache file in the memory pool.
[0138] The target pointer can refer to an array of pointers corresponding to the offset of the first cached file.
[0139] In one possible implementation, after determining the initial address corresponding to the target pointer in the target index tree based on the offset of the first cache file, the method further includes:
[0140] If the cache file corresponding to the initial address is not the second cache file, then obtain the second cache file from the file system;
[0141] Based on the second cache file, the cache file corresponding to the initial address is updated and replaced. After updating and replacing the cache file corresponding to the initial address, the initial address is determined to be the address of the second cache file in the memory pool.
[0142] In this application, the file system may refer to a pure user-space file system, including the libsysio module and the libfuse module, to enable interaction with the target node.
[0143] S204. Based on the address of the second cache file in the memory pool in the target index tree, perform read and write operations on the second cache file.
[0144] In one possible implementation, before performing read / write operations on the second cache file based on the address of the second cache file in the memory pool according to the target index tree, the method further includes:
[0145] Send a connection request to the memory management node in the global file cache system, and the memory management node will establish a connection with the target node after receiving the connection request;
[0146] After the connection between the memory management node and the target node is established, the metadata information of the memory nodes in the memory pool is obtained from the memory management node;
[0147] Based on the metadata information of the memory nodes in the memory pool, establish an RDMA connection with the memory nodes.
[0148] In this context, a memory management node refers to a node within the memory pool that manages the memory. Multiple memory management nodes can form a cluster, and software such as ZooKeeper can be used to ensure high availability and consistency. Each memory management node maintains metadata information about the memory nodes in the memory pool, including a list of memory nodes and information about each memory node in its internal table.
[0149] A connection request can refer to a connection request initiated by the target node to the memory management node, so that the memory management node can receive metadata information of the memory nodes in the memory pool.
[0150] RDMA connection refers to the RDMA connection between the target node and the memory node, enabling the target node to read and write data in the memory node.
[0151] In one possible implementation, the address of the second cache file in the memory pool is obtained based on the offset of the first cache file and the base address of the index tree corresponding to the name of the first cache file, including:
[0152] If the address of the second cache file in the memory pool is not found in the index tree, then the data of the second cache file in the file system is retrieved.
[0153] Based on the offset of the first cache file, determine the pointer corresponding to the second cache file in the index tree;
[0154] Based on the pointer corresponding to the second cache file in the index tree, the data of the second cache file is stored at the address corresponding to the pointer in the index tree, so that the address of the second cache file in the memory pool exists in the index tree.
[0155] Figure 3 A schematic diagram of the index tree structure provided in this application is shown below. Figure 3 As shown, the index tree can be a radix tree. The child nodes of the radix tree contain count, height, and slots fields. count represents the number of child nodes in the radix tree, height represents the height of the child node from the bottom layer to the current layer in the radix tree, and offset represents the offset of the page in the file. The offset is divided into bits, with 6 bits as a unit, which is used as the subscript of slots to locate the next child node, until the bottom layer is found. For example, when the target base address root of the target index tree has been determined through the index table, the offset of the first cache file is 64, and the offset is represented in binary as 000001000000. Starting from root, the pointer of the child node is determined as slots[0] based on bits 0-5 of the offset. slots[0] points to the first node of the second level of the index tree. The pointer of the child node is determined as slots[1] based on bits 6-11 of the offset. At this time, since height=1, it indicates that the query has ended. The position pointed to by the pointer slots[1] of the lowest level child node is the address of the second cache file in the memory pool.
[0156] Figure 4 A flowchart illustrating another file caching method provided in this application is shown below. Figure 4 As shown, the method includes:
[0157] S401. Construct memory nodes and memory management nodes in the memory pool. Memory nodes are connected to memory management nodes. Memory management nodes store a list of memory nodes corresponding to memory nodes.
[0158] S402. The compute node obtains the list of memory nodes in the memory management node in the memory pool and establishes an RDMA connection with all memory nodes in the memory pool.
[0159] S403. The memory management node finds the target memory node through a consistent hashing algorithm based on the memory allocation request, and returns the memory address of the memory node to the compute node, so that the compute node updates the cache file name and the memory address of the memory node to the index table.
[0160] S404. The compute node finds the target cached data in the index tree based on the memory address in the index table and the offset of the cached file.
[0161] Figure 5 A schematic diagram of the file caching device provided in this application is shown below. Figure 5 As shown, the file caching device 50 provided in this embodiment includes:
[0162] The acquisition module 501 is used to acquire a request for a cached file. The request for acquiring a cached file includes the target name of the first cached file and the offset of the first cached file.
[0163] The first determining module 502 is used to obtain the target base address from the index table according to the target name of the first cached file, wherein the target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree;
[0164] The second determining module 503 is used to determine the target index tree and the address of the second cache file in the memory pool based on the offset of the first cache file and the target base address, wherein the second cache file represents the block content of the first cache file;
[0165] The read / write module 504 is used to perform read / write operations on the second cache file based on the address of the second cache file in the memory pool in the target index tree.
[0166] The file caching device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0167] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.
[0168] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.
[0169] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0170] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0171] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0172] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0174] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0175] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0176] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0177] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0179] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0180] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0181] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0182] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A file caching method, characterized in that, The method, applied to a target node in a global file caching system, includes: A request to retrieve a cached file, wherein the request to retrieve a cached file includes the target name of the first cached file and the offset of the first cached file; Based on the target name of the first cached file, the target base address is obtained from the index table, wherein the target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree; Based on the offset of the first cache file and the target base address, the target index tree and the address of the second cache file in the target index tree in the memory pool are determined, wherein the second cache file represents the block content of the first cache file; Based on the address of the second cache file in the memory pool in the target index tree, read and write operations are performed on the second cache file.
2. The method according to claim 1, characterized in that, The method further includes: When the target base address is not obtained from the index table, the target memory node in the memory pool and the target base address of the target memory node are determined. Based on the target name of the first cached file, a target index tree corresponding to the target name is obtained, wherein the target index tree and the target base address have a corresponding relationship; Based on the target base address and the target name of the first cache file, the initial index table in the target node is updated to obtain the index table; According to the index table, the steps of obtaining the target base address from the index table based on the target name of the first cache file and performing read and write operations on the second cache file according to the address of the second cache file in the memory pool in the target index tree are executed.
3. The method according to claim 2, characterized in that, When the target base address is not obtained from the index table, the target memory node in the memory pool and the target base address of the target memory node are determined, including: When the target base address is not obtained from the index table, the name of the first cache file is hashed to obtain a hash value; Based on the hash value, the memory node list, and the encoding information of the memory nodes in the memory node list, the target memory node, the target number information of the target memory node, and the target base address of the target memory node are obtained, and the memory node list is a list of memory nodes in the memory pool.
4. The method according to claim 3, characterized in that, Based on the hash value, the memory node list, and the encoding information of the memory nodes in the memory node list, the target memory node, the target number information of the target memory node, and the target base address of the target memory node are obtained, including: Based on the encoding information of the memory nodes in the memory node list, determine the initial hash value corresponding to the encoding information of the memory nodes in the memory node list; Based on the hash value, determine the target hash value from the initial hash value; The target memory node is determined based on the target hash value; Based on the memory node list and the target memory node, the target number information of the target memory node and the target base address of the target memory node are obtained.
5. The method according to claim 1, characterized in that, The step of determining the target index tree and the address of the second cache file in the memory pool within the target index tree based on the offset of the first cache file and the target base address includes: The target index tree is determined based on the target base address; Based on the offset of the first cache file, determine the initial address corresponding to the target pointer in the target index tree; If the cache file corresponding to the initial address is the second cache file, then the initial address is determined to be the address of the second cache file in the memory pool.
6. The method according to claim 5, characterized in that, After determining the initial address corresponding to the target pointer in the target index tree based on the offset of the first cached file, the method further includes: If the cache file corresponding to the initial address is not the second cache file, then the second cache file is obtained from the file system; Based on the second cache file, the cache file corresponding to the initial address is updated and replaced, and after updating and replacing the cache file corresponding to the initial address, the initial address is determined to be the address of the second cache file in the memory pool.
7. The method according to claim 1, characterized in that, Before performing read / write operations on the second cache file based on its address in the memory pool within the target index tree, the method further includes: A connection request is sent to the memory management node in the global file caching system, and the memory management node establishes a connection with the target node after receiving the connection request. After the connection between the memory management node and the target node is established, the metadata information of the memory nodes in the memory pool is obtained from the memory management node; Based on the metadata information of the memory nodes in the memory pool, an RDMA connection is established with the memory nodes.
8. The method according to claim 1, characterized in that, The step of obtaining the address of the second cache file in the memory pool from the index tree based on the offset of the first cache file and the base address of the index tree corresponding to the name of the first cache file includes: If the address of the second cache file in the memory pool is not found in the index tree, then the data of the second cache file in the file system is obtained; Based on the offset of the first cache file, determine the pointer corresponding to the second cache file in the index tree; Based on the pointer corresponding to the second cache file in the index tree, the data of the second cache file is stored at the address corresponding to the pointer corresponding to the second cache file in the index tree, so that the address of the second cache file in the memory pool exists in the index tree.
9. A file caching device, characterized in that, include: The acquisition module is used to acquire a request for a cached file, wherein the request for acquiring a cached file includes the target name of the first cached file and the offset of the first cached file; The first determining module obtains the target base address from the index table based on the target name of the first cached file, wherein the target base address is the address of the target index tree, the target index tree is the index tree corresponding to the target name, the index table is a hash table, the key value in the index table represents the name of the cached file, the value value in the index table represents the base address of the index tree corresponding to the name of the cached file, and the index tree is a radix tree; The second determining module is used to determine the target index tree and the address of the second cache file in the memory pool based on the offset of the first cache file and the target base address, wherein the second cache file represents the block content of the first cache file; The read / write module is used to perform read / write operations on the second cache file according to the address of the second cache file in the memory pool in the target index tree.
10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.
12. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-8.