File reading method and related device

Abstract

Disclosed in embodiments of the present application are a file reading method and a related device. The method is applied to a computing node, the computing node is a computing node in a cluster system, the cluster system further comprises a storage system and a cache system, and the cache system is used for caching files in the storage system. The method comprises: acquiring a read request, the read request being used for reading a first file in the storage system; acquiring a storage path of the first file in the storage system from the read request; on the basis of a cache mapping table, determining whether the first file is stored in the cache system, the cache mapping table recording the correspondence between storage paths of files cached in the cache system in the storage system and storage paths of said files in the cache system; and when the first file is in the cache system, acquiring the storage path of the first file in the cache system from the cache mapping table, and reading the first file from the cache system. The method can reduce the operation burden of users, and improve the efficiency and accuracy of file storage management.

Description

A file reading method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 5, 2024, with application number 202410252264.X and application name “A file reading method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing, and in particular to a file reading method and related devices. Background Art

[0003] In high-performance computing (HPC) cluster systems, multiple computing nodes are interconnected through various interconnect technologies to form a high-speed computing computer cluster. These cluster systems are widely used in fields such as weather forecasting, computational simulation, and image processing. In HPC cluster systems, a large number of read-only files, such as executable files and dynamic library files, are typically stored in remote storage systems and accessed and loaded by the cluster's computing nodes during job execution.

[0004] To reduce the high concurrent throughput pressure on the storage system during cluster job execution, fast storage media is typically deployed on the compute nodes to create a cache system for frequently loaded read-only files. For example, the cache system might include a burst buffer (BB) file system. Users can use the BB file system to cache frequently used files when operating compute nodes.

[0005] However, this approach requires users to load files into the cache system in advance, increasing operational complexity. In addition, the cache system may cause unnecessary duplicate caching of the same file for different users. Summary of the Invention

[0006] The embodiments of the present application provide a file reading method and related devices for improving the efficiency and accuracy of file storage management in a computing cluster.

[0007] In a first aspect, an embodiment of the present application provides a file reading method, which is applied to a computing node, where the computing node is a computing node in a cluster system. The cluster system also includes a storage system and a cache system, where the cache system is used to cache files in the storage system. When the computing cluster receives a job to be processed, the scheduler assigns the job to the computing node. The method includes:

[0008] The computing node executes the job, obtains a read request in the job, and the read request is used to read the first file in the aforementioned storage system; analyzes and processes the parameters of the read request, and obtains the storage path of the first file in the read request in the storage system; then determines whether the first file is stored in the aforementioned cache system based on the cache mapping table, and the cache mapping table records the corresponding relationship between the storage path of the file cached in the cache system in the storage system and the storage path cached in the cache system; when the first file is in the cache system, the storage path of the first file in the cache system is obtained from the cache mapping table, and the first file is read from the cache system.

[0009] In this application, the storage system of the computing cluster stores executable files, dynamic library files, and other files required for the job, which are loaded and used by the computing nodes when the computing nodes execute the job. The cache system of the computing cluster can cache the files recently used by the computing node, and record the corresponding relationship between the storage path of the file in the storage system and the storage path cached in the cache system in the cache mapping table. When a read request to access the storage system is obtained, the access path of the read request is redirected to the cache system to obtain the corresponding cache file according to the cache mapping table.

[0010] Using this method, compute nodes only need to maintain a cache mapping table. When they receive a read request to the storage system, they redirect the request to the cache system. This eliminates the need for users to manage the cache system, reducing their operational burden and improving the efficiency and accuracy of file storage management. Using the cache mapping table to record the correspondence between the storage system's storage paths and the cache system's storage paths prevents duplicate caching of files, improving the cache system's usability.

[0011] In a possible implementation, obtaining the read request includes: detecting IO access loaded in the process, and intercepting the read request after detecting that the application in the computing node generates the read request.

[0012] In this application, processes accessing the storage system can be detected within a computing node. Specifically, for example, the mmap() function in the program code is used to map a file into memory. The mmap() function includes the file's storage path parameter. When a process is detected accessing the mmap() function, the process is intercepted and the storage path parameter is extracted.

[0013] By intercepting and analyzing read requests, this method can help understand system resource usage. When an application process detects a read operation on the storage system, it can redirect the access to a less-loaded burst buffer (BB) file system or a local file system, reducing the high-concurrency IO pressure on the storage system.

[0014] In a possible implementation, the storage path in the cache system is an access path of a file stored in a file system in the cache system.

[0015] In this application, the file system in the cache system is mainly used to cache files. Exemplary examples include a temporary file system (TMPFS) based on dynamic random access memory (DRAM), a disk file system based on a solid state drive (SSD) or a hard disk drive (HDD), such as the X file system (XFS), the fourth-generation extended file system (EXT4), the file allocation table file system (FATFS), the high-performance file system (HPFS), and the new technology file system (NTFS), or a BB file system connected to a computing node.

[0016] By using the above method, the files in the storage system are cached by the file system to obtain the access path in the file system. A mapping relationship between the storage path of the storage system and the storage path of the cache system (that is, the access path of the file system) can be constructed, so that access can be quickly redirected during access.

[0017] In a possible embodiment, the method further includes: when it is determined through the cache mapping table that the first file is not in the cache system, continuing to execute the read request to read the first file from the storage system, and caching the first file from the storage system to the cache system, and recording the mapping relationship between the storage path of the first file in the storage system and the storage path in the cache system in the cache mapping table.

[0018] This method eliminates the need for users to pre-load the files required for their jobs into the cache system. Instead, when a compute node first accesses a file in the storage system, it caches the file and records the corresponding mapping. This reduces the user's operational burden, improves system usability, and reduces frequent access to the storage system, speeding up file access.

[0019] In a possible implementation, the method further includes: recording the access frequency and / or the last access time of the first file in a cache mapping table, and determining whether the first file is eliminated from the cache system based on the frequency and / or the time.

[0020] By using this method, by recording file access frequency and / or the time of last access, the system can understand which files are frequently accessed ("hot files") and which files are inaccessible ("cold files") for a long time. This information allows for more intelligent cache management, ensuring that popular files remain in the cache, thereby improving cache hit rates and overall efficiency. It also allows for the removal of cold files to free up cache space for newer files.

[0021] In one possible implementation, the cache system includes a first cache system and a second cache system, the first cache system being a local cache system of a computing node, the second cache system being a shared cache system shared by computing nodes in a computing cluster, the first cache system corresponding to a first cache mapping table, and the second cache system corresponding to a second cache mapping table;

[0022] The method of determining whether the first file is stored in the cache system according to the cache mapping table includes: determining whether the first file is stored in the local cache system according to the first cache mapping table; if not, determining whether the first file is stored in the shared cache system according to the second cache mapping table.

[0023] Using this approach, we first check the local cache system to minimize data access latency and avoid data transfer across networks or storage systems. Secondly, we leverage the cache resources shared by multiple compute nodes to improve overall data availability. Using two cache mapping tables, we can effectively manage cache files in the cache system and avoid redundant caching.

[0024] In a possible implementation, the files cached by the cache system are read-only files.

[0025] In the second aspect, an embodiment of the present application provides a computing device, which is applied to a computing node, and the computing node is a computing node in a cluster system. The cluster system also includes a storage system and a cache system. The cache system is used to cache files in the storage system. The device includes: an acquisition module, which is used to obtain a read request, and the read request is used to read the first file in the storage system; the acquisition module is also used to obtain the storage path of the first file in the storage system in the read request; a determination module, which is used to determine whether the first file is stored in the cache system according to a cache mapping table, and the cache mapping table records the correspondence between the storage path of the file cached in the cache system and the storage path cached in the cache system; a processing module, which is used to obtain the storage path of the first file in the cache system from the cache mapping table when the first file is in the cache system, and read the first file from the cache system.

[0026] In a possible implementation, the acquisition module is specifically configured to: upon detecting a read request generated by an application in a computing node, intercept the read request.

[0027] In a possible implementation, the storage path in the cache system is an access path of a file in the file system.

[0028] In one possible embodiment, the processing module is also used to: when the first file is not in the cache system, continue to execute the read request to read the first file from the storage system, cache the first file from the storage system to the cache system, and record the mapping relationship between the storage path of the first file in the storage system and the storage path in the cache system in the cache mapping table.

[0029] In a possible implementation, the processing device is further configured to: record the access frequency and / or the last access time of the first file in a cache mapping table, and determine whether the first file is eliminated from the cache system based on the frequency and / or time.

[0030] In one possible implementation, the cache system includes a first cache system and a second cache system, the first cache system being a local cache system of a computing node, the second cache system being a shared cache system shared by computing nodes in a computing cluster, the first cache system corresponding to a first cache mapping table, and the second cache system corresponding to a second cache mapping table;

[0031] The determination module is specifically configured to: determine whether the first file is stored in the local cache system according to the first cache mapping table; if not, determine whether the first file is stored in the shared cache system according to the second cache mapping table.

[0032] In a possible implementation, the files cached by the cache system are read-only files.

[0033] In a third aspect, an embodiment of the present application provides a computing device, comprising a workflow processing apparatus comprising: a processor, a memory, and a transceiver. The memory stores a computer program or computer instructions, the processor is configured to call and execute the computer program or computer instructions stored in the memory, so that the processor implements the processing operations described in the first aspect and any one of the implementations of the first aspect, and the transceiver is configured to transmit and receive signals, such as implementing the receiving and transmitting operations described in the first aspect and any one of the implementations of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, the memory storing a computer program or computer instructions, and the processor being used to call and run the computer program or computer instructions stored in the memory so that the computing device cluster executes the above-mentioned first aspect and any optional method thereof.

[0035] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0037] In a seventh aspect, the present application provides a chip system comprising a processor for supporting an execution device or a training device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory for storing program instructions and data necessary for the execution device or the training device. The chip system may consist of a single chip or may include a chip and other discrete components.

[0038] As described above, the technical effects of the second, third, fourth, fifth and sixth aspects of this application can be understood in conjunction with the technical effects of the first aspect and any implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] FIG1 is a diagram of a network architecture applicable to an embodiment of the present application;

[0041] FIG2 is a schematic diagram of a flow chart of a file reading method provided in an embodiment of the present application;

[0042] FIG3 is a schematic diagram of a structure of a cache system provided in an embodiment of the present application;

[0043] FIG4 is another schematic diagram of the structure of the cache system provided in an embodiment of the present application;

[0044] FIG5 is another schematic diagram of the structure of the cache system provided in an embodiment of the present application;

[0045] FIG6 is a schematic diagram of a process for loading a file according to an embodiment of the present application;

[0046] FIG7 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0047] FIG8 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0048] FIG9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0050] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and in the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0051] The application scenarios involved in this application are application scenarios in which multiple computing nodes have high throughput or high concurrency for a data center, including but not limited to artificial intelligence (AI) scenarios and high performance computing (HPC) scenarios. The two application scenarios are explained below.

[0052] AI is a new technical discipline that studies the theories, methods, technologies, and application systems used to simulate, extend, and expand human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and more. The implementation of AI models requires vast amounts of data for training. This large amount of data and training tasks makes it difficult for a single device to independently train an AI model. Therefore, AI models are typically trained in distributed or centralized systems, enabling multiple computing devices to collaborate and train a shared AI model.

[0053] HPC uses efficient algorithms to rapidly complete data-intensive, compute-intensive, and input / output (I / O)-intensive computations. HPC is widely used in fields such as bioengineering, drug development, petroleum exploration, vehicle design (aerospace, shipbuilding, and automotive), materials engineering, nuclear explosion simulation, advanced weapons manufacturing, cryptography research, and various large-scale information processing applications.

[0054] Similar to AI systems, HPC achieves applications in various fields through cluster architecture, parallel algorithms, and software-related foundations. HPC clusters generally use parallel algorithms to divide a large computing business into multiple small jobs according to certain rules, and assign them to each computing node. The job processing results of each computing node are then merged into the final result of the large computing business. Since the calculation of these small jobs can generally be completed in parallel by multiple computing nodes, the problem processing time can be shortened. Specifically, the set of program instances required to complete a specific computing business usually corresponds to a set of processes, containers, or other runtime entities on one or more computers. In batch processing systems, jobs are also called "batch jobs."

[0055] Please refer to Figure 1, which is a schematic diagram of the network architecture of a cluster system 110 applicable to an embodiment of the present application. The cluster system 110 includes a computing cluster 111, a cache system 112 deployed on the computing cluster side, and a storage system 113 for storing read-only files.

[0056] Computing cluster 111 includes multiple computing nodes. In one possible implementation, cluster system 110 also includes a scheduler and multiple switches. After cluster system 110 receives a job from a client or service server, the scheduler schedules one or more computing nodes to execute the job. The switches connect multiple computing nodes to implement tasks such as data aggregation computing.

[0057] Illustratively, the computing nodes in computing cluster 111 may be implemented by devices with computing capabilities, such as servers. Specifically, the devices with computing capabilities may be provided with one or more processors, such as a central processing unit (CPU) or an application specific integrated circuit (ASIC), or a programmable logic device (PLD), which may be a complex programmable logical device (CPLD), an FPGA, a generic array logic (GAL), or any combination thereof.

[0058] Specifically, a computing node also includes memory and a network card. Cache refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, serving as temporary data storage for the operating system or other running programs. Cache includes at least two types of memory. For example, cache can be either random access memory (RAM) or read-only memory (ROM). For example, random access memory is dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a type of semiconductor memory that, like most random access memory (RAM), is a volatile memory device. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard drives, but has slower access speeds than DRAM and is also cheaper than DRAM. However, DRAM and SCM are merely exemplary in this embodiment; cache can also include other random access memories, such as static random access memory (SRAM). As for the read-only memory, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc. In actual applications, multiple caches and different types of caches can be configured in the computing node. This embodiment does not limit the number and type of caches. In addition, the cache can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the cache will not be lost. The cache with a power-saving function is called a non-volatile memory. The network card is used to communicate with the cache system 112 and the storage system 113.

[0059] Storage system 113 stores read-only files required by computing nodes when executing jobs, such as executable files, dynamic library files, and configuration files. These read-only files are stored in the disk array within storage system 113 and are frequently read by computing cluster 111 via a high-speed storage network, but are rarely or never modified.

[0060] Specifically, for example, in front-end simulation processing for electronic system design automation (EDA), a large amount of load is usually generated that requires running the same simulation tool. The executable files and dynamic libraries of the simulation tool need to be accessed and loaded multiple times in a short period of time. In addition, large-scale scientific simulation applications in HPC clusters usually require thousands or tens of thousands of computing nodes to be started simultaneously and access the same read-only file at the same time.

[0061] In response to the high concurrent throughput access pressure generated by the computing cluster 111 when executing jobs, a cache system 112, namely burst load storage (BB), is currently built near the computing cluster 111. The BB file system is used as a buffer between the front-end computing and the back-end storage, bridging the performance gap between the processing speed of the computing cluster 111 and the I / O bandwidth of the storage system 113. The BB file system is usually built with a high-performance storage device array, such as non-volatile random access memory (NVRAM) and solid-state drive (NVMe-SSD) based on the non-volatile memory protocol, which can provide an I / O bandwidth that is one to two orders of magnitude higher than that of the storage system 113. The computing nodes in the computing cluster 111 can improve the access performance of applications loading read-only files by accessing files of the BB file system.

[0062] It is worth noting that the cluster system 100 may further be configured with nodes having other functions, such as a management node configured to manage the computing nodes in the computing cluster 111. Furthermore, the architecture of the cluster system 100 shown in FIG1 is merely an example, and the present application may also be applied to other applicable cluster architectures.

[0063] However, when the BB file system is in use, it is necessary to pre-load the read-only files required for the job into the BB file system in advance and provide a new storage space name for the read-only file. For example, the access path of the read-only file "toola" in the storage system 113 is " / sharedfs / toola". Since a certain job requires multiple computing nodes of the computing cluster 111 to frequently access the read-only file when it is executed, the read-only file is stored in the BB file system, and the access path obtained in the BB file system is " / burst_buffer / toola". In addition, the access path of the read-only file needs to be updated on the code side so that the computing nodes of the computing cluster 111 can directly access the files in the BB file system when executing the job, so that there is no need to frequently load from the storage system 113. However, this will increase the complexity of the use of the entire system.

[0064] Based on this, please refer to Figure 2, which provides a file reading method for an embodiment of the present application. For a job submitted to the cluster system 110, the scheduler in the cluster system 110 allocates computing resources, namely computing nodes, to process the job.

[0065] 201. Obtain a storage path of the first file in the storage system from the read request;

[0066] In one possible implementation, an IO interceptor is deployed on the compute node, dynamically loaded via LD_PRELOAD, and executed in the application's process space. The IO interceptor's primary function is to detect and intercept read requests generated by the application, such as system call requests to read executable files or dynamic library files.

[0067] Specifically, when an application is detected executing the mmap() function, which is used to map a file or other object into memory, the application obtains the file access path parameter in the mmap() function. This access path parameter is the storage path of the first file in the storage system. For example, if the application code accesses a file in the storage system using mmap(" / sharedfs / toola",...), the storage path of the first file is " / sharedfs / toola".

[0068] Furthermore, the mmap() function also includes other parameters, such as whether the file is a read-only file. In actual applications, the aforementioned IO interceptor can achieve accurate access path interception of the access path of the storage system in the read request.

[0069] It should be noted that in addition to the mmap() function, actual applications may also include more function methods for reading storage systems, but all of them will set parameters for accessing the file storage path, or have methods such as setting the default access path and appending a specific file name. The IO interceptor can set the corresponding interception method based on the specific function methods used in the program code, which is not limited here.

[0070] 202. Determine whether the first file is stored in the cache system according to the cache mapping table;

[0071] After obtaining the storage path of the first file, the computing node uses the cache mapping table to determine whether there is an access path to a file in the file system of the cache system that corresponds to the storage path of the first file. Specifically, the computing node queries the cache system's file system to determine whether the first file exists. If the first file exists in the cache system, step 203 is executed; if not, step 204 is executed.

[0072] Optionally, the cache mapping table is a local cache table of the computing node; the cache system can be the local cache of the computing node, that is, the local file system of the computing node, including a memory file system (temporary file system, TMPFS) based on dynamic random access memory (DRAM) and swap partition, a disk file system based on a solid state drive (SSD) or a hard disk drive (HDD), such as a log file system (X file system, XFS), a fourth-generation extended file system (EXT4), a file allocation table file system (FATFS), a high-performance file system (HPFS), and a new technology file system (NTFS).

[0073] Please refer to FIG3 , which is a schematic diagram of the structure of the cache system as a local cache of a computing node in an embodiment of the present application.

[0074] The file "toola" and the file "toolb" are stored in the storage system (sharedfs). In the computing cluster 111, the file "toola" is stored in the local cache of computing node A. In one possible implementation, the file system used by the local cache of computing node A is TMPFS, and the access path of the file "toola" in the local cache is " / cnA / tmpfs / toola", and the file "toola" in the local cache is the cache file of the file "toola" in the storage system. Correspondingly, the local cache table of computing node A records the mapping relationship between the storage path of the file "toola" in the storage system and the storage path in the local cache as " / sharedfs / toola= / cnA / tmpfs / toola". Among them, the local cache table can be stored in the local file system of computing node A or in a database system connected to computing node A, and the specific details are not limited here.

[0075] Specifically, computing node A obtains the first file storage path " / sharedfs / toola" in step 201, and determines based on the local cache table that the storage path of the first file in the cache system is " / cnA / tmpfs / toola".

[0076] It is understandable that the local cache of computing node B can also use TMPFS to store the file "toolb". The local cache table of computing node B records the mapping relationship between the storage path of the file "toolb" in the local cache of computing node B and the storage path of the storage system as " / sharedfs / toolb= / cnB / tmpfs / toolb".

[0077] In an embodiment of the present application, the computing node implements the redirection function for application access by managing its local file system and maintaining a local mapping table. In addition, when the computing node loads read-only files of an external cache system such as executable files and dynamic library files, a memory mapping (mmap) method is usually adopted to map the files or other objects directly to the memory of the computing node. In this process, the computing node needs to maintain key information including the mapping start address, size, access rights, etc., which are all stored in the kernel space. The external cache system needs to include kernel state components to support these operations. However, when the computing node redirects the access path to the local file system, there is no need to consider complex issues such as kernel state.

[0078] It should be noted that the structural diagram shown in Figure 3 is only a possible situation. In actual applications, the local cache of the computing node may include multiple levels of different file systems. As shown in Figure 4, computing node A includes a TMPFS file system and an XFS file system, and can cache the file "toolb" in the XFS file system. Correspondingly, the local cache table of computing node A records the mapping relationship " / sharedfs / toolb = / cnA / xfs / toolb" corresponding to the file "toolb". In addition, there may be more other file systems, and each file system may store multiple different files, which are not limited here.

[0079] Optionally, the cache mapping table is a local cache table of a computing node and a shared cache table shared by multiple computing nodes in a computing cluster; the cache system may also be a cache system 112 including a local cache of a computing node connected to multiple computing nodes in a computing cluster 111, i.e., a BB file system.

[0080] Please refer to Figure 5, which is a structural diagram of a cache system including a local cache of a computing node and a burst buffer (BB) connected to multiple computing nodes in an embodiment of the present application. In the storage system (sharedfs), the files "toola", "toolb" and "toolc" are stored. The file "toolc" is stored in the file system of BB. The second file information includes the access path " / burst_buffer / toolc" of the file "toolc" in the BB file system. Correspondingly, the shared cache table records the mapping relationship between the storage path of the file "toolc" in the storage system and the storage path in the BB file system as " / sharedfs / toolc= / burst_buffer / toolc". Among them, the shared cache table can be stored in the BB file system or in a database system connected to each computing node of the computing cluster 111, and the specific details are not limited here.

[0081] Specifically, computing node A obtains the storage path of the first file as " / sharedfs / toolc" in step 201. At this point, computing node A does not obtain the corresponding cache information through the local cache table. It then continues to query the shared cache table and determines that the storage path of the first file in the BB file system is " / burst_buffer / toolc".

[0082] 203. Get the first file from the cache system.

[0083] After determining the storage path of the first file in the cache system through the cache mapping table, the computing node obtains the first file from the cache system.

[0084] Please refer to FIG6 , which is a schematic diagram of the process of loading files using a multi-level file storage system.

[0085] After the IO interceptor of the computing node obtains the access request of the job, it modifies the file access path parameter in the mmap() function to the storage path of the first file in the cache system, and redirects the first file to the local file system or the BB file system.

[0086] Specifically, the IO interceptor on compute node A obtains the code "mmap(" / sharedfs / toola",...)" for job "job1" and, by querying the cache mapping table of the cache system, redirects the file access to "mmap(" / cnA / tmpfs / toola",...)" and then reads the file "toola". The IO interceptor on compute node B obtains the code "mmap(" / sharedfs / toolc",...)" for job "job2" and, by querying the cache mapping table of the cache system, redirects the file access to "mmap(" / burst_buffer / toolc",...)" and then reads the file "toolc".

[0087] It is understandable that if the cache information is determined by the local cache table, the first file is obtained from the local file system of the computing node; if the cache information is determined by the shared cache table, the first file is obtained from the BB file system.

[0088] Based on this, in one possible implementation, when multiple jobs are simultaneously submitted to cluster system 110 for processing, for example, including jobs "job3" and "job4," both jobs "job3" and "job4" need to load the file "toold" during execution. When the file "toold" is cached in the BB file system, "job3" and "job4," even if running on different computing nodes, can directly obtain the corresponding second file information from the shared cache table and redirect the file.

[0089] It should be noted that in the embodiment of the present application, there is no need to cache the required files in the cache system in advance. Instead, the cache mapping table is managed by the computing node to achieve separation of cache data and cache control, and the cache data is stored using the existing multi-level storage system. Exemplarily, the cache system for the computing node includes: a DRAM-based memory file system (such as TMPFS) as a first-level file cache, a disk file system (such as XFS) based on SSD or HDD as a second-level file cache, and a BB file system connected to multiple computing nodes as a third-level file cache. Among them, the file read and write speed of the high-level file cache is higher than the file read and write speed of the low-level file cache. The cache mapping table records the storage path of the files required by the computing node in the file system. When the file is migrated due to the elimination policy, the storage path recorded in the cache mapping table must be updated accordingly.

[0090] Alternatively, for example, if the file "toole" is initially stored in TMPFS, the local cache table records the storage path as " / cnA / tmpfs / toole". However, due to various factors, such as decreased access frequency or insufficient storage space, the file "toole" is migrated from TMPFS to XFS, and the storage path becomes " / cnA / xfs / toole". At this point, the information recorded in the local cache table must also be updated accordingly.

[0091] Furthermore, when a file is migrated from the local file system of a computing node to the BB file system, the record of the file in the local cache table will be deleted, and at the same time, a corresponding record will be added to the shared cache table to ensure data consistency.

[0092] In addition, there may be more levels of file cache for storage in the cache system, which are not limited here.

[0093] Furthermore, the computing node may also record the access frequency and access time of each file in the file storage system. The access frequency is the number of processes that simultaneously access the file, and the access time is the last time the file was accessed by a process. For example, the access frequency and access time may be recorded in a cache mapping table containing the file information, or in other log files, the specific details of which are not limited herein.

[0094] Specifically, after redirecting the file path accessed by mmap() to the cache system, the access frequency and access time corresponding to the first file are updated, and the storage location of each file in the multi-level file storage system is updated according to the cache elimination strategy.

[0095] Optionally, if a least recently used (LRU) policy is used, files that have not been accessed in a recent period of time may be migrated to a next-level file cache based on the recorded access time.

[0096] Optionally, if a least frequently used (LFU) policy is used, files with less frequent access may be migrated to a next level of file cache based on the recorded access frequencies.

[0097] It is understandable that when the storage space of the last level file cache in the file storage system is insufficient, the files to be eliminated are directly deleted. In actual applications, there are more cache elimination strategies, which only require migrating files that meet the strategy results to the next level file cache, and the specific ones are not limited here.

[0098] In addition, files in the cache system with an access frequency lower than a first threshold may be periodically cleared, for example, files in the cache system with an access frequency of 0 may be periodically deleted.

[0099] 204. Obtain the first file from the storage system, cache the first file in the cache system, and record it in the cache mapping table.

[0100] When in step 202, the computing node does not find that the first file is cached in the cache system through the cache mapping table, it obtains the first file from the storage system, caches the first file in the cache system, and records the storage path of the first file in the storage system and the storage path in the cache system in the cache mapping table.

[0101] Furthermore, the access frequency and access time corresponding to the second file may be created in the cache mapping table.

[0102] In an embodiment of the present application, a computing node maintains a list of access paths for a file (i.e., a cache mapping table), which can ensure that the files required for the job are quickly located. The user does not need to manage the files in the cache system, which reduces the user's operational burden and improves the efficiency and accuracy of file storage management. When a computing node needs to access a file, it first checks whether the file exists in the cache system. If the file is in the cache system, it is read directly from the cache system, reducing the access delay to the remote storage system. If the file is not in the cache system, the file is pulled from the storage system according to the storage path and added to the cache system. Furthermore, the local cache space of the computing node is utilized to simplify the complexity of the entire cache system without increasing storage costs. And the cache mapping table is used to record the correspondence between the storage path of the storage system on the application code side and the storage path of the cache system, which can avoid duplicate caching of files and improve the usability and efficiency of the cache system. For a BB file system connected to multiple computing nodes, through a shared cache mapping table, files in the BB file system will not be cached repeatedly, saving storage space of the BB file system and improving the consistency and synchronization of files.

[0103] The above describes the file reading method provided by the embodiment of the present application. The following describes the computing device provided by the embodiment of the present application. Please refer to Figure 7, which is a schematic diagram of the structure of the computing device of the embodiment of the present application. Computing device 700 can be used to perform the steps performed in the embodiments shown in Figures 2 to 6. Please refer to the relevant description of the above method embodiment for details.

[0104] The computing device 700 includes an acquisition module 701, a determination module 702, and a processing module 703. The acquisition module 701 can implement corresponding communication functions, and the determination module 702 and the processing module 703 are used for data processing. The acquisition module 701 can also be called a communication interface or a communication unit.

[0105] The computing device 700 can be used to perform the actions in the above method embodiment. The acquisition module 701 is used to perform the reception-related operations in the above method embodiment, and the determination module 702 and the processing module 703 are used to perform the processing-related operations in the above method embodiment.

[0106] The computing device 700 includes:

[0107] An acquisition module 701 is configured to acquire a read request, where the read request is used to read a first file in the storage system;

[0108] The acquisition module 701 is further configured to acquire a storage path of the first file in the read request in the storage system;

[0109] a determination module 702 configured to determine whether the first file is stored in the cache system according to a cache mapping table, wherein the cache mapping table records a correspondence between a storage path of a file cached in the cache system in the storage system and a storage path of the file cached in the cache system;

[0110] The processing module 703 is configured to obtain a storage path of the first file in the cache system from the cache mapping table when the first file is in the cache system, and read the first file from the cache system.

[0111] In a possible implementation, the acquisition module 701 is specifically configured to: upon detecting a read request generated by an application in a computing node, intercept the read request.

[0112] In a possible implementation, the storage path in the cache system is an access path of a file in the file system.

[0113] In one possible embodiment, the processing module 702 is also used to: when the first file is not in the cache system, continue to execute the read request to read the first file from the storage system, cache the first file from the storage system to the cache system, and record the mapping relationship between the storage path of the first file in the storage system and the storage path in the cache system in the cache mapping table.

[0114] In a possible implementation, the processing device 703 is further configured to: record the access frequency and / or the last access time of the first file in the cache mapping table, and determine whether the first file is eliminated from the cache system based on the frequency and / or time.

[0115] In one possible implementation, the cache system includes a first cache system and a second cache system, the first cache system being a local cache system of a computing node, the second cache system being a shared cache system shared by computing nodes in a computing cluster, the first cache system corresponding to a first cache mapping table, and the second cache system corresponding to a second cache mapping table;

[0116] The determination module 702 is specifically configured to: determine whether the first file is stored in the local cache system according to the first cache mapping table; if not, determine whether the first file is stored in the shared cache system according to the second cache mapping table.

[0117] In a possible implementation, the files cached by the cache system are read-only files.

[0118] The determination module 702 and processing module 703 in the above embodiment can be implemented by at least one processor or processor-related circuits. The acquisition module 701 can be implemented by a transceiver or transceiver-related circuits. The acquisition module 701 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0119] This application also provides a computing device 100. As shown in FIG8 , computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0120] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 shows a single bus line, but this does not imply a single bus or type of bus. Bus 104 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0121] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0122] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0123] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module, determination module, and processing module, thereby implementing the file reading method. That is, the memory 106 stores instructions for executing the file reading method.

[0124] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned computing device, thereby implementing the file reading method. That is, the memory 106 stores instructions for executing the file reading method.

[0125] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0126] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0127] As shown in Figure 9, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the file reading method.

[0128] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a file reading method.

[0129] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the file reading method.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A file reading method, characterized in that: The method is applied to a computing node, which is a computing node in a cluster system. The cluster system also includes a storage system and a cache system. The cache system is used to cache files in the storage system. The method includes: Obtaining a read request, where the read request is used to read a first file in the storage system; Obtaining a storage path of the first file in the read request in the storage system; determining whether the first file is stored in the cache system according to a cache mapping table, wherein the cache mapping table records a correspondence between a storage path of a file cached in the cache system in the storage system and a storage path of the file cached in the cache system; When the first file is in the cache system, the storage path of the first file in the cache system is obtained from the cache mapping table, and the first file is read from the cache system.

2. The method according to claim 1, characterized in that The obtaining the read request comprises: When the read request generated by the application in the computing node is detected, the read request is intercepted.

3. The method according to claim 1 or 2, characterized in that The storage path in the cache system is the access path of the file in the file system.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When the first file is not in the cache system, the read request is continued to be executed to read the first file from the storage system, and the first file is cached from the storage system to the cache system, and the mapping relationship between the storage path of the first file in the storage system and the storage path in the cache system is recorded in the cache mapping table.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The access frequency and / or the last access time of the first file are recorded in the cache mapping table, and whether the first file is eliminated from the cache system is determined based on the frequency and / or the time.

6. The method according to any one of claims 1 to 5, characterized in that The cache system includes a first cache system and a second cache system, the first cache system is a local cache system of the computing node, the second cache system is a shared cache system shared by the computing nodes in the computing cluster, the first cache system corresponds to a first cache mapping table, and the second cache system corresponds to a second cache mapping table; The determining, according to the cache mapping table, whether the first file is stored in the cache system includes: determining, according to the first cache mapping table, whether the first file is stored in the local cache system; If not, determining whether the first file is stored in the shared cache system according to the second cache mapping table.

7. The method according to any one of claims 1 to 6, characterized in that The files cached by the cache system are read-only files.

8. A computing device, characterized in that The apparatus includes at least one processor coupled to a memory; The memory is used to store programs or instructions; The at least one processor is configured to execute the program or instruction so as to enable the apparatus to implement the method according to any one of claims 1 to 7.

9. A computer program product comprising instructions, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device, cause the computer to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed on a computer device, the computer device executes the method according to any one of claims 1 to 7.

Images