File access method and device and computing cluster

Abstract

The embodiment of the invention provides a file access method and device and a computing cluster, and relates to the technical field of computers. The file access method is applied to the computing cluster, the computing cluster comprises a management node and a plurality of computing groups, the computing nodes in each computing group are fully connected with storage nodes, and the management node receives jobs and allocates service execution nodes for the jobs from the computing nodes; the management node establishes a virtual file system for the job and sends the virtual file system to the service execution node; when the service execution node writes in a first file of a job, generating a virtual path of the first file in a virtual file system, and mapping the virtual path to an actual path of a file in a local file system of the service execution node, and writing the first file in a storage position in a storage node which is indicated by the actual path and is located in the same calculation group as the service execution node. According to the invention, efficiency and convenience of file access can be considered.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a file access method, device, and computing cluster. Background Art

[0002] A job in a computing cluster refers to the execution of an application launched in the cluster, and can include all processes initiated by that application. Large-scale jobs, such as particle simulations and meteorological simulations, often involve high-concurrency file access within a short period of time.

[0003] In related technologies, computing nodes and storage nodes in a computing cluster can be grouped, and computing nodes and storage nodes in the same group can be fully connected. On this basis, the code of the application to which the job belongs can be added with a code that perceives the grouping situation. In this way, when the computing nodes in the group execute the job, they can determine the storage nodes in the group through the above-mentioned code that perceives the grouping situation, and quickly access the storage nodes in the group through full connection, thereby coping with high-concurrency file access in a short period of time. Among them, the code logic of the application to which the job belongs can be added with access logic that adapts to the specific implementation of the file system.

[0004] However, adding grouping-aware code to the application code requires prior knowledge of the computing cluster's grouping and modification of the application code, which severely reduces the convenience of file access. Summary of the Invention

[0005] To solve the above technical problems, the present application provides a file access method, device, and computing cluster. The file access method can achieve fast file access within the group by mapping the virtual path under the virtual file system established by the management node for the job to the actual path of the virtual path under the local file system of the execution node of the job, indicating the actual path of the storage location in the storage node in the same group as the execution node, thereby achieving fast file access without the application to which the job belongs being aware of the group, and taking into account the convenience and performance of file access.

[0006] In a first aspect, an embodiment of the present application provides a file access method, which is applied to a computing cluster, wherein the computing cluster includes a management node, multiple computing nodes, and multiple storage nodes, the multiple computing nodes and the multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected. The method includes: the management node receives a job and assigns a business execution node that executes the job to the job from multiple computing nodes; the management node establishes a virtual file system for the job and sends the virtual file system to the execution node; when the business execution node writes a first file in the process of executing the job, a virtual path for writing the first file is generated under the virtual file system, and the virtual path is mapped to the actual path of the file established under the local file system of the business execution node, and the first file is written according to the location indicated by the actual path, and the location indicated by the actual path is the storage location in the storage node in the same computing group as the business execution node.

[0007] In an embodiment of the present application, a management node can create a virtual file system for a job and send it to the job's business execution node. When writing the first file of the job, the business execution node can then map the virtual path of the first file in the virtual file system to the actual path of the file established in the local file system of the business execution node. The actual path indicates a storage location in a storage node in the same group as the execution node. In this way, when the business execution node writes the first file to the storage location indicated by the actual path, file access can be guaranteed within the computing group of the business execution node, thereby achieving fast file access without the application to which the job belongs being aware of the group, thus balancing the convenience and performance of file access.

[0008] According to the first aspect, before mapping the virtual path to the actual path of the file established under the local file system of the business execution node, the file access method provided by the embodiment of the present application may also include: the business execution node pre-establishes multiple empty files under the local file system; accordingly, mapping the virtual path to the actual path established under the local file system of the business execution node may include: mapping the virtual path to the actual path of the first empty file among the multiple empty files under the local file system, and allocating a storage location for the first empty file, and the allocated storage location is the location indicated by the actual path of the first empty file.

[0009] In an embodiment of the present application, the business execution node pre-establishes multiple empty files in the local file system, that is, creates a file pool. In this way, during the operation process, the file system can establish a mapping relationship between the actual path of the first empty file in the file pool and the first file written, and allocate a storage location for the first empty file. The allocated storage location is the location indicated by the actual path of the first empty file. In this way, the empty files in the file pool can be bound to the virtual path to achieve the effect of creating a file. Subsequently, file management can be carried out with the help of this binding relationship without the need to actually operate on the files, which is more efficient and realizes the asynchronous creation of files, thereby further improving the performance of file access.

[0010] According to the first aspect, or any implementation of the first aspect above, after writing the first file according to the location indicated by the actual path, the method also includes: the business execution node receives a deletion request for the first file; deleting the virtual path of the first file in the virtual file system according to the deletion request, and releasing the mapping relationship between the virtual path of the first file and the actual path of the file established in the local file system.

[0011] In the embodiment of the present application, file deletion can be achieved by changing the mapping relationship between the virtual path and the actual path, without the need to perform actual operations on the file, which more efficiently realizes the asynchronous creation of file, thereby further improving the performance of file access.

[0012] According to the first aspect, or any implementation of the first aspect above, mapping the virtual path to the actual path established under the local file system of the business execution node includes: mapping the virtual path to the actual path established under the local file system of the business execution node according to a mapping rule adapted to the job, the mapping rule including one or more of the following rules: the access time of the file under the directory depth of the actual path is less than or equal to the access time of the file under the directory depth of the virtual path; the access time of the file under the directory width of the actual path is less than or equal to the access time of the file under the directory width of the virtual path; the number of files indicated by the actual path is greater than or equal to the number of first files; when the first file exceeds the upper limit of the single file size under the local file system, there are multiple second files under the actual path that do not exceed the size upper limit; when the number of first files is multiple and the total size of the multiple first files is equal to or less than the upper limit of the single file size under the local file system, there is one second file under the actual path; the total capacity of the storage space indicated by each actual path corresponding to the job is less than or equal to the storage space quota of the job; the total number of files under all actual paths corresponding to the job is less than or equal to the file number quota of the job.

[0013] In the embodiments of the present application, a mapping relationship between a virtual path and an actual path is determined by predefined file mapping rules adapted to the access requirements of a job, thereby ensuring that file access under this mapping relationship better meets job requirements and further improving file access performance. For example, this can meet job access performance requirements, rationally utilize storage resources, meet different job mode requirements, and ensure quality of service (QoS) for file access.

[0014] According to the first aspect, or any implementation of the first aspect above, a business execution node for executing the job is assigned to the job from multiple computing nodes, including: determining the number of computing groups for the job based on the I / O bandwidth requirement of the job and the I / O bandwidth upper limit of the computing group; and assigning a business execution node for executing the job from the computing nodes of the number of computing groups.

[0015] In an embodiment of the present application, a computing group corresponds to an I / O bandwidth upper limit. When the I / O bandwidth upper limit is less than the I / O bandwidth requirement of the job, multiple computing groups that meet the I / O bandwidth requirement can be allocated to the job, thereby meeting the I / O bandwidth requirement of the job.

[0016] According to the first aspect, or any implementation of the first aspect above, multiple storage nodes constitute a BurstBuffer cache system, the computing group constitutes a BurstBuffer group, and the local file system is a BurstBuffer file system.

[0017] The embodiment of the present application can shield the specific implementation of the file system from the application program in a scenario where the file system is a Burst Buffer file system, thereby ensuring that the application program does not need to pay attention to the BB grouping situation of the file system to execute the business, and can reduce the difficulty of the job to access the file.

[0018] In the second aspect, an embodiment of the present application provides a file access device, which is applied to a computing cluster, wherein the computing cluster includes a management node, multiple computing nodes, and multiple storage nodes. The multiple computing nodes and multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected. The device includes a management module and a business execution module; when the management module and the business execution module are respectively running on the management node and the computing node, the management node and the computing node execute the instructions of the method in the first aspect or any possible implementation of the first aspect.

[0019] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0020] In a third aspect, an embodiment of the present application provides a computing device cluster, comprising: a management node, multiple computing nodes, and multiple storage nodes, wherein the multiple computing nodes and the multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected; the management node and the computing nodes are used to execute instructions of the method in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed in a computing device cluster, enables the computing device cluster to execute instructions of the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 FIG. 1 is a schematic diagram illustrating a process of file access by a job under a Burst Buffer grouping architecture;

[0024] Figure 2 This is one of the structural diagrams of a computing cluster provided in an embodiment of the present application;

[0025] Figure 3 This is one of the structural diagrams of a computing cluster provided in an embodiment of the present application;

[0026] Figure 4 This is a software architecture diagram of a computing cluster provided in an embodiment of the present application;

[0027] Figure 5 This is a software architecture diagram of a file system provided in an embodiment of the present application;

[0028] Figure 6 This is a flowchart of a file access method provided by an embodiment of the present application;

[0029] Figure 7is an example diagram of the virtual path and the actual path provided in the embodiment of the present application;

[0030] Figure 8 This is a schematic diagram of file access by a job under the Burst Buffer grouping architecture provided in an embodiment of the present application;

[0031] Figure 9 This is a structural block diagram of a file access device provided in an embodiment of the present application;

[0032] Figure 10 is a structural block diagram of a computing device 100 provided in an embodiment of the present application;

[0033] Figure 11 This is one of the structural block diagrams of a computing device cluster provided in an embodiment of the present application;

[0034] Figure 12 This is one of the structural block diagrams of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0037] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0038] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0039] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0040] First, some of the terms and related technologies involved in this application are explained to facilitate understanding.

[0041] High-performance computing (HPC) refers to a complete computer system capable of achieving a certain level of computing power. HPC requires the collaboration of multiple central processing units (CPUs) or multiple hosts (e.g., multiple computers). HPC is primarily used to rapidly complete data-intensive, compute-intensive, and I / O-intensive calculations in fields such as scientific research, engineering design, finance, industry, and social management. Typical applications include bioengineering, new drug development, petroleum exploration, vehicle design (aerospace, ships, and automobiles), materials engineering, cutting-edge equipment manufacturing, cryptography, and various types of large-scale information processing.

[0042] Burst Buffer (BB): A near-computing I / O system for large-scale supercomputing systems. It can support tens of thousands of compute nodes, efficiently utilize the available I / O bandwidth of near-computing storage, and provide near-linear performance scalability. Used to accelerate jobs in supercomputing, Burst Buffer nodes are located between compute nodes and the storage nodes that serve as the capacity layer, acting as storage nodes for the performance layer. Burst buffer nodes are typically constructed from arrays of high-performance storage devices, such as non-volatile random access memory (NVRAM) and / or solid-state drives (SSDs). Burst buffer nodes can provide higher I / O bandwidth (e.g., one to two orders of magnitude higher) than storage nodes that serve as the capacity layer.

[0043] A job is a single execution of an application started by the scheduler. In a distributed cluster, an application job includes the application and all processes started by the application at runtime. After an application is started, it can run one or more jobs.

[0044] Scheduler: Responsible for job management functions such as the management and allocation of computing resources in the computing cluster, and the lifecycle of computing jobs. Examples of schedulers include: Slurm (Simple Linux Utility for Resource Management, an open-source, fault-tolerant, and highly scalable Linux cluster supercomputing system resource management and job scheduling system) in the HPC field; LSF (Load Sharing Facility, a distributed cluster management system software responsible for computing resource management and batch job scheduling. It provides users with a unified cluster resource access interface, allowing users to transparently access the entire cluster resource); Donau (a core component of the high-performance computing software stack, whose main function is to manage user cluster resources, connect to user businesses, and schedule the computing tasks generated by user businesses to appropriate cluster resources according to certain rules); YARN (a software package manager) in the big data field, etc.

[0045] NVMe: Non-Volatile Memory Express, a non-volatile memory host controller interface specification with single-disk read and write bandwidth reaching GB levels. Devices using the NVMe protocol (such as NVRAM) are primarily used as storage devices for high-performance, near-computing, high-speed file systems in Burst Buffer systems.

[0046] The following describes the file access process for a job using a file system that uses the Burst Buffer grouping architecture as an example.

[0047] For example, Figure 1 FIG. 1 is a schematic diagram showing the process of file access by a job under the Burst Buffer grouping architecture. Figure 1As shown, Burst Buffers typically use a grouping approach to support the high-performance I / O required by large-scale computing clusters. Specifically, compute nodes and back-end (BB) nodes are divided into multiple BB groups (e.g., BB Group 1, BB Group 2, ..., BB Group X). Each BB group includes at least one compute node and at least one BB node. The size of a BB group can be determined based on the aggregate bandwidth provided by the Burst Buffer system, the specifications of the managed NVMe devices, the specifications of the SSDs, the network bandwidth between the compute nodes and the Burst Buffer, and the network topology formed by the BB groups. For example, BB groups can have sizes of 32 nodes, 64 nodes, 128 nodes, or 256 nodes. BB groups communicate directly within each other via fully interconnected connections (e.g., a bus), while inter-BB groups communicate using a forwarding mechanism, also known as a component forwarding mechanism. I / O read and write performance within a BB group can reach 90GB. However, inter-BB group I / O read and write performance is affected by the forwarding mechanism and network topology, resulting in lower efficiency.

[0048] The inter-group forwarding mechanism mainly involves that when a job running on a compute node accesses a file that is not in the BB group to which the compute node belongs, the compute node uses the forwarding service to access each BB group in the computing cluster across groups until the access is successful. In other words, inter-group access is unordered. However, the size of each BB group is limited, and large jobs often require multiple BB groups to support them. Figure 1 , the compute node belongs to BB Group 1. The job running on this compute node includes Process 1 under the job and Process 2 under the job. The files accessed by the job are distributed on the BB nodes within BB Group 1 to BB Group X. Process 1 under the job can directly access files on the BB nodes in BB Group 1 through the full connection within the group, and access files on the BB nodes in BB Group 2 through the inter-group forwarding mechanism. Process 2 under the job accesses files on the BB nodes in BB Group 2 to BB Group X through the inter-group forwarding mechanism. In other words, large jobs are prone to a large number of inter-group accesses, which also means a large amount of disordered file access, resulting in a serious degradation of file access performance.

[0049] In one example, an application can be aware of the BB grouping and prioritize file read and write operations within the BB grouping where the application's job resides, reducing cross-BB grouping and achieving better file access performance. However, this requires integrating code within the application's logic to understand the file system architecture, such as BB grouping, which significantly increases the cost and barrier to entry for adaptation. Furthermore, if the file system architecture changes, such as a change in the BB grouping topology, the application's BB grouping-aware code will need to be modified accordingly, further reducing file access convenience.

[0050] The embodiment of the present application provides a file access method to solve the above-mentioned problem. The method can establish a virtual file system for the job through the management node and send it to the business execution node of the job. Then, when the business execution node writes the first file of the job, it can map the virtual path of the first file under the virtual file system to the actual path of the file established under the local file system of the business execution node. The actual path indicates the storage location in the storage node of the same group as the execution node. In this way, the business execution node writes the first file according to the storage location indicated by the actual path to ensure file access within the computing group of the business execution node, thereby achieving fast file access without the application to which the job belongs being aware of the group, taking into account both the convenience and performance of file access.

[0051] Before describing the technical solution of the embodiment of the present application, the operating platform of the file access method of the embodiment of the present application is first described with reference to the accompanying drawings. The file access method provided by the embodiment of the present application can be applied to a computing cluster.

[0052] In one example, Figure 2 This is one of the structural diagrams of a computing cluster provided in the embodiment of this application. Figure 2As shown, the computing cluster 200 may include multiple coupled service nodes, such as service node 201, service node 202, ..., service node 20n, where n is the number of service nodes. In other words, the computing cluster 200 is a network comprising multiple nodes, and each node may include one or more service nodes. In specific applications, the service nodes may be computing devices (such as servers), virtual machines, containers, and other devices that can provide computing resources and / or storage resources. The service nodes that provide computing resources may be referred to as computing nodes, and the nodes that provide storage resources may be referred to as storage nodes. When the storage nodes constitute the BurstBuffer cache system, the storage nodes may be, for example, SSDs. In this case, the SSDs may be included in the physical device on which the computing nodes are deployed, or the SSDs may be deployed in physical devices different from the physical devices. When the service nodes are physical devices, the coupling method between different service nodes may be a local area network, which may include communication devices such as switches and network cards. This application does not limit the specific form of the service nodes in the computing cluster.

[0053] In specific applications, the service nodes (i.e., compute nodes) in the computing cluster that provide computing resources can initiate application jobs, and the service nodes (i.e., storage nodes) in the computing cluster that provide storage resources can store job files. Applications can, for example, be programs for implementing particle physics simulation, weather forecasting, terrain analysis, image processing, fluid dynamics modeling, and other services.

[0054] In one example, Figure 3 This is one of the structural diagrams of a computing cluster provided in the embodiment of this application. Figure 3 As shown, the computing cluster may include a login node 301, a management node 302 communicatively connected to the login node, multiple computing nodes 303 communicatively connected to the management node, and multiple storage nodes communicatively connected to the multiple computing nodes. Each storage node includes multiple performance tier storage nodes 3041 and multiple capacity tier storage nodes 3042, and the performance tier storage nodes 3041 and the capacity tier storage nodes 3042 are communicatively connected.

[0055] The login node 301 is used to provide application login services to users through command line (CLI), portal website, etc.

[0056] The management node 302 is used to manage jobs through the manager, for example, receiving and submitting jobs for application execution, allocating computing nodes and storage nodes to jobs, and establishing a virtual file system for jobs. It will be appreciated that the above management can be performed based on management instructions sent by the login node 301.

[0057] Compute node group 303 is used to run jobs. A compute node group can include one or more compute nodes. Different jobs can run on different or the same compute nodes or compute node groups. For example, the compute group running job JOB1 is group 1, and the compute group running job JOB2 is group 2.

[0058] Performance tier storage node group 3041 provides higher-performance file access. For example, it can be composed of multiple Burst Buffer nodes aggregated with multiple SSDs. Performance tier storage nodes use a BB grouping architecture: multiple Burst Buffer nodes and compute nodes are divided into one or more compute groups, such as compute group 1 and compute group 2. Compute nodes running job JOB1 are grouped into compute group 1, and compute nodes running job JOB2 are grouped into compute group 2. Compute nodes and storage nodes within a compute group are fully connected, and each compute node in the group can access every storage node within the group.

[0059] The capacity layer storage node 3042 is used to provide a large-capacity storage space, and may be, for example, a node including multiple SSDs and multiple traditional hard disk drives (HDDs).

[0060] It should be understood that Figure 2 and Figure 3 The computing cluster shown is only one example of a computing cluster, and a computing cluster may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 2 and Figure 3 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0061] For example, Figure 4 This is a software architecture diagram of a computing cluster provided by an embodiment of the present application. Figure 4 As shown, the computing cluster 400 may include a scheduler 401 and a file system 402. In one example, the file access implemented by the file system 402 is integrated with the file submission (staging) process implemented by the scheduler 401. In this case, the operation of the scheduler 401 may include three I / O stages: pre-processing (Stage in), computing processing, and post-processing (Stage out). The scheduler 401 creates a service instance of the file system 402 in the pre-processing stage, accesses files through the service instance of the file system 402 in the computing processing stage for job execution, and enters the post-processing stage after the job execution is completed, performing operations such as cleaning up the service instance of the file system 402, cleaning up files, and solidifying data.

[0062] It should be understood that Figure 4 The software architecture of the computing cluster shown is only one example of the software architecture of the computing cluster, and the software architecture of the computing cluster may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 4 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0063] For example, Figure 5 This is a software architecture diagram of a file system provided by an embodiment of the present application. Figure 5 As shown, the file system provided by the embodiment of the present application can shield the specific implementation of the file system from the job, thereby ensuring that file access from the job perspective is carried out in accordance with the business logic of the job itself, and can be regarded as a job-level file system. The file system includes but is not limited to the following modules: service management module 501, namespace management module 502, namespace high availability management module 503, file system bridge API 504 and file system adapter interface 505. The file system can be integrated with the scheduler and can be started during the job declaration cycle, that is, when the job is started, to provide file access services for the job during the job operation. The job-level file system can run on an existing file system in a computing cluster (such as a Burst Buffer file system). Specifically:

[0064] The service management module 501 is used to connect with the scheduler and start the service instance of the file system for a specific job in the pre-processing and post-processing stages of file access according to the instructions of the scheduler, thereby managing the service instance of the file system through service instance creation, service instance monitoring and service instance recovery, and exiting the file system through service instance deletion.

[0065] The path management module 502 is used to map the virtual path of the file accessed by the job with the actual path of the file actually stored in the Burst Buffer node through the mapping relationship between the virtual path and the actual path, the file pool and the shared file, and perform file access. Among them, the virtual path is the access path of the file in the virtual file system established for the job by the service management module 501 when the job writes the file. The virtual path is sent to the namespace management module 502 by the service management module 501. The actual path is the namespace management module 502.

[0066] Path high availability management module 503 is used to ensure high availability of the file system during job execution. For example, if a node failure causes the related processes to die, this module is responsible for failure recovery after recovery. It is understood that path high availability management module 503 can be an existing file system in a computing cluster running a job-level file system, such as a module provided by a burst buffer file system, a parallel file system service (PFS), or an object storage service (OBS).

[0067] The file system bridge API 504 is used to provide an interface for using the job-level file system of the present application embodiment. It can be integrated with standard I / O interfaces such as POSIX (Portable Operating System Interface) and MPI (Message Passing Interface). In this way, the application code does not need to be modified to adapt to the interface and can continue to use the standard I / O interface, further improving the convenience of file access.

[0068] The file system adapter interface 505 is used to provide an adaptation layer between other file systems running the job-level file system provided by the embodiment of the present application and the job-level file system, so as to realize the interaction between other file systems and the job-level file system. Among them, other file systems can be, for example, Burst Buffer, PFS, OBS and other file systems. In this way, it is convenient to switch to different file systems according to the needs of the application, reuse the functions of other file systems, etc., and further improve the convenience of file access.

[0069] It should be understood that Figure 5 The software architecture of the file system shown is merely one example of the software architecture of the file system, and the software architecture of the file system may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 5 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0070] The following combination Figures 6 to 8 The file access method provided in the embodiment of the present application is described in detail.

[0071] For example, Figure 6 This is a flowchart of a file access method provided by an embodiment of the present application. Figure 6As shown, the file access method can be applied to a computing cluster, which includes a management node, multiple computing nodes, and multiple storage nodes. The multiple computing nodes and the multiple storage nodes are divided into multiple computing groups. The computing nodes and storage nodes in each computing group are fully connected. The method may include:

[0072] S601: A management node receives a job and assigns a business execution node to execute the job from multiple computing nodes.

[0073] The jobs received by the management node are jobs for launching applications on the computing cluster. These jobs can be one or more. The management node uses a scheduler to assign one or more compute nodes, also known as service execution nodes, from the multiple compute nodes in the computing cluster to the jobs. The service execution nodes are responsible for executing the corresponding jobs.

[0074] In specific applications, the management node can allocate computing nodes for jobs based on the file system architecture and current resource conditions. For example, see Figure 3 , the computing nodes in computing group group 2 have no available computing resources, and the computing nodes in computing group group 1 can be allocated to job JOB1.

[0075] Optionally, allocating a business execution node to execute the job from multiple computing nodes may include:

[0076] Determine the number of computing groups for the job based on the job's I / O bandwidth requirements and the computing group's I / O bandwidth limit.

[0077] Assign a business execution node for the job from the computing nodes in the computing groups.

[0078] For example, each group has its own maximum bandwidth, for example, 10 Gbps. If a job requires 20 Gbps, if one group cannot meet the bandwidth requirement, it can be split into two groups to meet the access bandwidth needs of different jobs. In other words, each compute group has an I / O bandwidth cap. If the I / O bandwidth cap is less than the job's I / O bandwidth requirement, multiple compute groups can be allocated to the job to meet the I / O bandwidth requirement, thereby meeting the job's I / O bandwidth requirement.

[0079] In an optional implementation, the above-mentioned multiple storage nodes constitute a Burst Buffer cache system, and accordingly, the computing group constitutes a Burst Buffer group, and the local file system is a Burst Buffer file system.

[0080] For example, after the manager completes the allocation of business execution nodes, the management node can obtain the computing group to which the job belongs. Figure 3Computing group group 1 includes the computing nodes in computing node group 303 that runs job JOB1 and the BB nodes in performance layer storage node group 3041 that are divided into group 1, and BB group 2 includes the computing nodes in computing node group 303 that runs job JOB2 and the BB nodes in performance layer storage node group 3041 that are divided into group 2.

[0081] S602: The management node creates a virtual file system for the job and sends the virtual file system to the execution node.

[0082] For example, Figure 7 This is an example diagram of the virtual path and actual path provided by the embodiment of the present application. Figure 7 As shown, the virtual file system is a job-level file system that does not need to consider the specific circumstances of the computing group. For example, the virtual file system can be organized with metadata in a B+ tree structure or a hash structure. A B+ tree is a tree data structure, an n-ary tree, where each node typically has multiple child nodes. A B+ tree consists of a root node, internal nodes, and leaf nodes. The root node may be a leaf node or a node with two or more child nodes. A hash structure is a technique or process that uses a hash function to map keys (k) and values ​​(v) into a hash table, allowing for faster access to elements.

[0083] S603: When the business execution node writes the first file during the execution of the job, a virtual path for writing the first file is generated under the virtual file system, and the virtual path is mapped to the actual path of the file established under the local file system of the business execution node, and the first file is written according to the location indicated by the actual path. The location indicated by the actual path is a storage location in a storage node located in the same computing group as the business execution node.

[0084] Still see Figure 7 The file system is a Burst Buffer grouping architecture. Both virtual paths and real paths can be considered as access paths to files written / read during job execution. The difference is that the virtual path is the access path to files in the virtual file system, which allows jobs to focus on business logic management of files without having to be aware of grouping. For example, the virtual path is Figure 7 The path under the metadata organization of the B+ tree structure is . / JOB1, and the path under the metadata organization of the hash structure is . / JOB2. The actual path indicates the actual storage location of the file in the storage node of the computing group, facilitating faster intra-group access to files by computing node. For example, the actual path is . / file11 and . / file21, the actual paths of the files created in the local file system of all BB groups, i.e., the business execution nodes.

[0085] In addition, the embodiments of the present application do not limit the specific forms of the virtual path and the actual path, and can be set according to application requirements. For example, the virtual path and the actual path can be in the form of a path description (such as C: / ...), a block storage form (such as Block X), etc.

[0086] For example, Figure 8 Schematic diagram of file access by jobs under the burst buffer grouping architecture provided by the embodiment of the present application. Figure 8 As shown, the process under the job running on the computing node initiates a write request to the virtual path, and then the computing node generates a virtual path for writing the first file under the virtual file system, and maps the virtual path to the actual path of the file established under the local file system of the business execution node, and writes the first file according to the location indicated by the actual path, that is, writes the file to the storage location of the storage node of the BB group where the computing node is located. In this way, the specific implementation of the file system can be shielded from the process under the job, while taking into account the effect of access within the group. Figure 5 Take this as an example to further explain the file access process:

[0087] like Figure 5 As shown, file access can include two main processes: management flow and data flow.

[0088] The management flow is embedded in the job lifecycle by the local file system on the compute node and the scheduler on the management node. It mainly includes two stages: pre-processing and post-processing. Pre-processing includes but is not limited to the following processes:

[0089] When a job is started, the scheduler will allocate computing nodes and computing resources in the allocated computing nodes to the job based on the real-time resource situation; the local file system obtains the node information and resource topology of the job from the scheduler, and starts the job-level file system service instance on the computing node of the job. Figure 5 The service instance of the file system shown; after the job-level file system service instance is started, localization preprocessing is performed, which at least includes the above S602.

[0090] Optionally, the localization preprocessing may further include the following processing: before mapping the virtual path to the actual path of the file established in the local file system of the service execution node, the file access method provided by the embodiment of the present application may further include:

[0091] The business execution node pre-creates multiple empty files in the local file system;

[0092] Accordingly, mapping the virtual path to the actual path established in the local file system of the service execution node may include:

[0093] The virtual path is mapped to the actual path of the first empty file among the multiple empty files in the local file system, and a storage location is allocated to the first empty file, where the allocated storage location is the location indicated by the actual path of the first empty file.

[0094] For example, Figure 5 As shown, the file system creates multiple empty files, namely file pools, for the job in the preprocessing stage. In this way, during the job running process, the file system can use the actual path of the file in the file pool as the actual path in the mapping relationship between the virtual path and the actual path of the job, thereby binding the files in the file pool to the virtual path to achieve the effect of creating files. Subsequently, the file management can be carried out with the help of this binding relationship without the need to perform actual operations on the files, which is more efficient and realizes the asynchronous creation of files, thereby further improving the performance of file access.

[0095] After the preprocessing is completed, the job-level path management service instance can provide file access services based on the mapping relationship between the virtual path and the actual path during job execution.

[0096] In an optional implementation manner, the mapping of the virtual path to the actual path established in the local file system of the service execution node may specifically include:

[0097] Map the virtual path to the actual path established in the local file system of the service execution node according to the mapping rules adapted to the job. The mapping rules include one or more of the following rules:

[0098] Mapping rule 1: The access time of the file at the directory depth of the actual path is less than or equal to the access time of the file at the directory depth of the virtual path;

[0099] For example, the directory rule is to optimize the directory depth, for example, mapping multiple folders under the virtual path to as few folders as possible under the actual path. Figure 7 The directory depth of the virtual path of JOB2 is greater than the directory depth of the actual path of Group 2. In this way, the business execution node can access the file faster when actually accessing it in the storage node. In one implementation, the file system can determine the directory depth of the first actual path in the mapping relationship based on a depth threshold that matches the type, scale and other attribute information of the job and the directory depth of the actual path virtual path. That is to say, based on Rule 1, the directory depth of the first actual path can also be less than, greater than or equal to the depth threshold to ensure that the directory of the first actual path is more reasonable. The specific size relationship can be set according to application requirements, and the embodiments of the present application do not limit this.

[0100] Mapping rule 2: The file access time under the directory width of the actual path is less than or equal to the file access time under the directory width of the virtual path.

[0101] For example, this rule is also called directory width optimization, for example, mapping a folder at a certain level in the virtual path to multiple folders at the same level in the actual path. Or, mapping multiple folders at a certain level in the virtual path to one folder in the actual path.

[0102] Mapping rule 3: the number of files indicated by the actual path is greater than or equal to the number of the first file;

[0103] Exemplarily, mapping rule 3 is adapted to the I / O mode of the job, that is, there are different file layouts for different I / O modes to meet different access requirements, further improving the performance of file access. Specifically, N is the number of concurrent processes that execute a task of the application, and M is the number of files accessed by the process. The I / O mode is N:M. The I / O mode is N:1 mode, and a single file under the virtual path can be split into multiple files under the actual path. In this case, one virtual path can be mapped to multiple actual paths. Multiple files under the virtual path can be split into more files under the actual path, or the same number of files as the actual path. In this case, N virtual paths can be mapped to M actual paths, and N=M is ​​a positive integer. In this way, the business execution node can access multiple actual paths through multiple concurrent processes to improve file access efficiency.

[0104] Mapping rule 4: When the first file exceeds the upper limit of a single file size in the local file system, there are multiple second files in the actual path that do not exceed the upper limit;

[0105] When there are multiple first files and the total size of the multiple first files is equal to or smaller than the upper limit of the size of a single file in the local file system, there is one second file in the actual path;

[0106] For example, if a first file exceeds the upper limit for a single file in the local file system, the presence of multiple second files in the actual path that do not exceed the upper limit can split the first file into multiple second files, thereby overcoming the local file system's restrictions on oversized files. If there are multiple first files and the total size of the multiple first files is equal to or less than the upper limit for a single file in the local file system, the presence of a second file in the actual path can combine the multiple first files into a single second file, thereby aggregating small files, reducing the waste of storage resources, and reducing the complexity of the actual path.

[0107] Mapping rule 5: The total capacity of the storage space indicated by each actual path corresponding to the job is less than or equal to the storage space quota of the job;

[0108] Exemplarily, this rule can achieve job storage quota guarantee in quality of service (QoS) guarantee: different jobs have different storage space quotas, thereby providing differentiated service capability guarantees for different users. In addition, the file method provided in the embodiment of the present application, after S603, can also include: when the total storage space of the access location indicated by all actual paths in the mapping relationship of the file system is greater than the space quota corresponding to the job, an error prompt is output. In this way, the problem of excessive job preemption of resources leading to a decline in the performance of the entire computing cluster can be avoided, and the performance of file access can be further improved.

[0109] Mapping rule 6: The total number of files in all actual paths corresponding to the job is less than or equal to the file quota of the job;

[0110] For example, this rule can achieve the guarantee of the number of job files in QoS guarantee, ensure more reasonable file system management efficiency, and provide differentiated service capabilities for different users.

[0111] Optionally, the service execution node receives the access request of the job through a standard I / O interface.

[0112] For example, see Figure 5 The local file system interface provided by the embodiments of the present application complies with the specifications of the standard I / O interface. For example, the local file system interface provided by the embodiments of the present application is integrated with standard I / O interfaces such as POSIX (Portable Operating System Interface) and MPI (Message Passing Interface). In this way, the local file system receives access requests from jobs through the standard I / O interface, and the application code does not need to be modified to adapt to the interface and can continue to use the standard I / O interface, thereby further improving the convenience of file access.

[0113] In an optional embodiment, the data structure of each local file system in the mapping relationship is adapted to the file mapping rule satisfied by the actual path;

[0114] Accordingly, accessing files in the actual path may include:

[0115] Access the files in the actual path through a retrieval method that adapts to the data structure of the local file system.

[0116] For example, Figure 5As shown, similar to how local file systems can use different data structures to store files, for local file systems: for local file systems with deep directory hierarchies, a hash-structured metadata organization, also known as KV storage, can be used; for file systems with frequent file traversal operations, a B+ tree-structured metadata organization can be used. Accordingly, when accessing files under the actual path in KV storage, V can be obtained through K, improving access efficiency for files with relatively deep hierarchical depths. When accessing files under the actual path in a B+ tree structure, B+ tree traversal can be performed as concurrently as possible, improving access efficiency for files with relatively wide hierarchical widths.

[0117] The embodiment of the present application achieves file retrieval optimization: based on the mapping rules satisfied by the actual path in the mapping relationship, the local file system is used to access the file using a data structure that is suitable for the mapping rules. Accordingly, the file is accessed using a retrieval method that is compatible with the data structure, thereby further improving access performance.

[0118] In an optional example, after writing the first file according to the location indicated by the actual path, the file access method provided by the embodiment of the present application may further include:

[0119] The service execution node receives a deletion request for the first file;

[0120] The virtual path of the first file in the virtual file system is deleted according to the deletion request, and the mapping relationship between the virtual path of the first file and the actual path of the file established in the local file system is released.

[0121] For example, see Figure 7 , the storage location of the first file under the virtual file system is the location indicated by the virtual path . / JOB1, and the storage location of the first file under the local file system of the business execution node is the actual path . / file11. If the job requests to delete the first file, then the business execution node of the job can delete the virtual path . / JOB1, so that the file deletion is realized from the job perspective. Moreover, by releasing the mapping relationship between the virtual path . / JOB1 and the actual path . / file11, the storage resources indicated by the actual path . / file11 can be released. In an optional example, if the actual path . / file11 is the path of an empty file created by the business execution node in the preprocessing stage, the mapping relationship between the virtual path . / JOB1 and the actual path . / file11 can also release the empty file for continued use. In this way, the deletion of the file can be achieved by changing the mapping relationship between the virtual path and the actual path, without the need to perform actual operations on the file, which is more efficient and realizes the asynchronous creation of file, thereby further improving the performance of file access.

[0122] Optionally, the file access method provided in the embodiment of the present application may further include:

[0123] When a job is finished, the business execution node cleans up or stores the files in the file pool based on the attribute information of each file accessed by the job.

[0124] Still see Figure 5 After the job completes, the business execution node performs post-processing through the job-level file system service instance. For example, if the file's attributes indicate that the file is cached data, the service node may clean up the file pool; if the file's attributes indicate that the file is non-cached data, the service node may persist the file data. After post-processing is complete, the job-level file system service instance exits. This ensures that the data of the terminated job is available for subsequent use and avoids resource waste.

[0125] The data flow during the file access process described above represents the mapping service provided by the job-level namespace management service instance during job execution. This primarily involves the physical layout of the file, accomplished during the namespace mapping process in S602. After the mapping is complete, the job can run according to the application's native logic. Accordingly, the file system responds to file access requests during the job execution, performing file access, i.e., reading and writing files.

[0126] This application also provides a file access device. For example, Figure 9 This is a structural block diagram of a file access device provided by an embodiment of the present application. Figure 9 As shown, the file access device can be applied to a computing cluster, which includes a management node, multiple computing nodes, and multiple storage nodes. The multiple computing nodes and multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected. The device includes: a management module and a service execution module, the management module runs on the management node, and the service execution module runs on the computing node;

[0127] The management module is used to receive jobs and assign a business execution node for the jobs from multiple computing nodes; establish a virtual file system for the jobs and send the virtual file system to the execution node;

[0128] The business execution module management module is used to generate a virtual path for writing the first file under the virtual file system when writing the first file during the execution of the job, and map the virtual path to the actual path of the file established under the local file system of the business execution node, and write the first file according to the location indicated by the actual path. The location indicated by the actual path is the storage location business execution module in the storage node located in the same computing group as the business execution node.

[0129] The management module and the service execution module can be implemented in software or hardware. For example, the implementation of the management module will be described below using the management module as an example. Similarly, the implementation of the management module and the service execution module can refer to the implementation of the management module.

[0130] As an example of a software functional unit, a module may include a code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the management module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0131] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0132] As an example of a hardware functional unit, a management module may include at least one computing device, such as a server. Alternatively, the management module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0133] The multiple computing devices included in the management module can be distributed in the same region or in different regions. They can also be distributed in the same AZ or in different AZs. Similarly, they can be distributed in the same VPC or across multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0134] It should be noted that in other embodiments, the management module can be used to execute any step in the file access method, the management module can be used to execute any step in the file access method, and the business execution module can be used to execute any step in the file access method. The steps that the management module and the business execution module are responsible for implementing can be specified as needed. The full functions of the computing node are realized by respectively implementing different steps in the file access method through the management module and the business execution module.

[0135] The present application also provides a computing device 100. Exemplarily, Figure 10 This is a block diagram of a computing device 100 provided in an embodiment of the present application. Figure 10 As shown, computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0136] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, memory 106, processor 104, communication interface 108).

[0137] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0138] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0139] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned management module and service execution module, thereby implementing the file access method. That is, the memory 106 stores instructions for executing the file access method.

[0140] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0141] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0142] For example, Figure 11 This is one of the structural diagrams of a computing device cluster provided in the embodiment of this application. Figure 11 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the file access method.

[0143] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the file access method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the file access method.

[0144] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, each used to perform part of the functions of the computing node. In other words, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules of the management module and the service execution module.

[0145] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. For example, Figure 12 This is one of the structural diagrams of a computing device cluster provided in the embodiment of this application. Figure 12 As shown, in one possible implementation, two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is achieved through a communication interface in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the management module. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of the service execution module.

[0146] Figure 12 The connection method between the computing device clusters shown can be based on the consideration that the file access method provided in this application requires a large amount of reading of resource data, parsing of resource data, and sorting of resource data, so it is considered to entrust the functions implemented by the business execution module to the computing device 100B for execution.

[0147] It should be understood that Figure 12 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0148] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the computer program product causes the at least one computing device to execute a file access method, or a file access method.

[0149] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a file access method, or instruct the computing device to execute a file access method.

[0150] Among them, the computing device, computing device cluster, computer storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0151] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0152] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A file access method, characterized in that: Applied to a computing cluster, the computing cluster includes a management node, multiple computing nodes, and multiple storage nodes, the multiple computing nodes and the multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected, the method includes: The management node receives the job and allocates a service execution node for executing the job from the plurality of computing nodes; The management node establishes a virtual file system for the job and sends the virtual file system to the business execution node; When the business execution node writes the first file in the process of executing the job, a virtual path for writing the first file is generated under the virtual file system, and the virtual path is mapped to the actual path of the file established under the local file system of the business execution node, and the first file is written according to the location indicated by the actual path. The location indicated by the actual path is a storage location in a storage node located in the same computing group as the business execution node.

2. The method according to claim 1, characterized in that After writing the first file according to the location indicated by the actual path, the method further includes: The service execution node transmits the mapping relationship between the virtual path and the actual path to the management node, and updates the virtual file system in the management node; When receiving the read request for the first file, the management node reads the first file according to the mapping relationship.

3. The method according to claim 1 or 2, characterized in that Before mapping the virtual path to the actual path of the file established in the local file system of the service execution node, the method further includes: The service execution node pre-creates a plurality of empty files in the local file system; Mapping the virtual path to an actual path established in the local file system of the service execution node includes: The virtual path is mapped to the actual path of the first empty file among the multiple empty files in the local file system, and a storage location is allocated for the first empty file, where the allocated storage location is the location indicated by the actual path of the first empty file.

4. The method according to any one of claims 1 to 3, characterized in that After writing the first file according to the location indicated by the actual path, the method further includes: The service execution node receives a deletion request for the first file; The virtual path of the first file in the virtual file system is deleted according to the deletion request, and the mapping relationship between the virtual path of the first file and the actual path of the file established in the local file system is released.

5. The method according to any one of claims 1 to 4, characterized in that Mapping the virtual path to an actual path established in the local file system of the service execution node includes: Map the virtual path to an actual path established in the local file system of the service execution node according to a mapping rule adapted to the job, wherein the mapping rule includes one or more of the following rules: The access time of the file at the directory depth of the actual path is less than or equal to the access time of the file at the directory depth of the virtual path; The access time of the file under the directory width of the actual path is less than or equal to the access time of the file under the directory width of the virtual path; The number of files indicated by the actual path is greater than or equal to the number of the first files; When the first file exceeds the upper limit of a single file size in the local file system, there are multiple second files under the actual path that do not exceed the upper limit of size; When there are multiple first files and the total size of the multiple first files is equal to or smaller than the upper limit of the size of a single file in the local file system, there is one second file in the actual path; The total capacity of the storage space indicated by each actual path corresponding to the job is less than or equal to the storage space quota of the job; The total number of files in all actual paths corresponding to the job is less than or equal to the file quantity quota of the job.

6. The method according to any one of claims 1 to 5, characterized in that The allocating a service execution node for executing the job from the plurality of computing nodes includes: Determining the number of computing groups for the job based on the I / O bandwidth requirement of the job and the I / O bandwidth upper limit of the computing group; Allocate a service execution node for executing the job from the computing nodes of the plurality of computing groups.

7. The method according to claim 1, characterized in that The multiple storage nodes constitute a Burst Buffer cache system, the computing group constitutes a Burst Buffer group, and the local file system is a Burst Buffer file system.

8. A file access device, characterized in that: Applied to a computing cluster, the computing cluster includes a management node, multiple computing nodes, and multiple storage nodes, the multiple computing nodes and the multiple storage nodes are divided into multiple computing groups, the computing nodes and storage nodes in each computing group are fully connected, and the device includes a management module and a service execution module; When the management module and the service execution module are respectively run on the management node and the computing node, the management node and the computing node execute the method according to any one of claims 1 to 7.

9. A computing cluster, characterized in that: include: A management node, multiple computing nodes, and multiple storage nodes, wherein the multiple computing nodes and the multiple storage nodes are divided into multiple computing groups, and the computing nodes and storage nodes in each computing group are fully connected; The management node and the computing node are configured to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program, which, when executed in a computing device cluster, causes the computing device cluster to execute the method according to any one of claims 1 to 7.

Images