Multi-domain metadata management method for high performance computing scenarios

Abstract

The application discloses a multi-domain metadata management method for a high-performance computing scene, and is applied to a distributed file system, and the method comprises the following steps: a multi-domain metadata register acquires and saves a workflow name, a storage node list file and a file system priority configuration file of each temporary file system; the multi-domain metadata register waits for and acquires a view fusion request of the temporary file system needing to be fused, and reads the corresponding storage node list file and the file system priority configuration file according to the workflow name of the temporary file system needing to be fused to perform view fusion, so as to provide a unified metadata view. The application dynamically constructs a metadata view by using a differentiated method, decouples consistency constraints among each other, flexibly adjusts a management strategy of metadata according to different requirements, reduces global synchronization management overhead and index overhead of metadata, and further effectively improves concurrent access performance of metadata, and consumption of network and storage resources is reduced.

Description

Technical Field

[0001] The present invention relates to file system data management technology, and in particular to a multi-domain metadata management method for high-performance computing scenarios. Background Art

[0002] In high-performance computing (HPC) operations, there is a large demand for concurrent access to metadata. However, the cost of implementing global synchronization management and maintenance of metadata is very high. On the one hand, it occupies valuable storage capacity and introduces unnecessary storage performance overhead. On the other hand, frequent synchronization operations reduce the overall efficiency of the file system. Therefore, how to decouple the consistency constraints between each other and improve the concurrent access performance of metadata is of great significance for metadata management in high-concurrency scenarios. To optimize metadata performance, existing methods mainly include the following three methods:

[0003] One approach is to use a partial consistency model. This approach, based on the partitioning of metadata by application, ensures consistency within each application without performing global synchronization across all applications. Although this approach reduces the frequency of global synchronization and, in turn, the system's synchronization overhead, it cannot guarantee global data consistency. This can lead to data inconsistencies in multi-application scenarios, impacting the overall system reliability and data accuracy.

[0004] The second approach is to design an ordered index layout to reduce compression. This approach reduces compression operations and optimizes read performance through read clustering and spatial locality, effectively improving read efficiency and reducing performance loss during metadata access. However, this approach relies on the orderliness of data and the spatial locality of access to optimize performance. Therefore, it may not be effective when data access patterns are variable or data distribution is uneven. Furthermore, designing a complex index layout may increase system implementation complexity and maintenance costs.

[0005] The third is to use client logging and snapshot methods to avoid global synchronization. By performing logging and generating snapshots on the client, the dependence on global metadata synchronization is reduced, thereby reducing synchronization overhead and improving the system's concurrent processing capabilities. Since the client logging and snapshot methods require maintaining a large amount of log and snapshot data on the client device, this may take up a lot of storage space and increase management complexity.

[0006] In summary, existing methods can optimize metadata performance in some aspects, but their respective limitations may also affect the overall efficiency and reliability of the system in certain specific scenarios. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to reduce the global synchronization management overhead and indexing overhead of metadata.

[0008] To address the above-mentioned problems in the existing technology, a multi-domain metadata management method for high-performance computing scenarios is provided. The method uses differentiated methods to dynamically construct metadata views, decouples mutual consistency constraints, flexibly adjusts metadata management strategies according to different needs, reduces the global synchronization management overhead and indexing overhead of metadata, and thus effectively improves the concurrent access performance of metadata and reduces the consumption of network and storage resources.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A multi-domain metadata management method for high-performance computing scenarios is applied to a distributed file system, wherein the distributed file system includes a multi-domain metadata register and multiple temporary file systems. The method includes:

[0011] The multi-domain metadata registrar obtains and saves the workflow name, storage node list file and file system priority configuration file of each temporary file system;

[0012] The multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system that needs to be merged, and reads the corresponding storage node list file and file system priority configuration file according to the workflow name of the temporary file system that needs to be merged to perform view fusion to provide a unified metadata view.

[0013] Furthermore, the temporary file system includes a temporary file system server and a temporary file system client. Before the multi-domain metadata register obtains and saves the workflow name, storage node list file, and file system priority configuration file of each temporary file system, it includes:

[0014] The temporary file system client obtains the configuration file of the multi-domain metadata register and parses it to obtain the address of the multi-domain metadata register, and then obtains the specified communication protocol variable in the network;

[0015] The temporary file system server retrieves the configuration information of the temporary file system and sends it to the temporary file system client;

[0016] The temporary file system client sends a request specifying a communication protocol to the multi-domain metadata registrar.

[0017] Furthermore, after the temporary file system server retrieves the configuration information of the temporary file system and sends it to the temporary file system client, the method further includes: if the temporary file system client is used for the first time, the temporary file system client sends the configuration information of the temporary file system to the multi-domain metadata registrar through a specified communication protocol.

[0018] Furthermore, before the multi-domain metadata registrar waits for and obtains a view fusion request of the temporary file system to be fused, the following steps are performed:

[0019] In the specified compute node, set the fusion identity environment variable to the specified value;

[0020] The temporary file system client reads the fusion identification environment variable of the computing node to which the temporary file system belongs, and sends a view fusion request to the multi-domain metadata registrar if the fusion identification environment variable is a specified value.

[0021] Furthermore, when performing view fusion by reading the corresponding storage node list file and the file system priority configuration file according to the workflow name of the temporary file system to be fused, the following steps are included:

[0022] According to the workflow name of the temporary file system that needs to be merged, the corresponding storage node list file and file system priority configuration file content are read cyclically, and spliced ​​into a new storage node list file and file system priority configuration file, and then the new storage node list file and file system priority configuration file are sent to the temporary file system that needs to be merged.

[0023] Furthermore, after reading the corresponding storage node list file and the file system priority configuration file according to the workflow name of the temporary file system to be merged to perform view fusion to provide a unified metadata view, the following steps are also included:

[0024] If a file is written into the metadata view, determining whether the file exists in the temporary file system corresponding to the metadata view;

[0025] If the file already exists and only exists in a temporary file system, directly save the file to the temporary file system;

[0026] If the file exists in multiple temporary file systems, select a corresponding temporary file system to save the file according to the file system priority in the temporary file system configuration information;

[0027] If the file does not exist in any temporary file system, it is determined whether the computing node of the metadata view has a local file system. If so, the local file system is first selected to save the file. Then, the hash value of the file path is calculated and the corresponding temporary file system is selected to save the file through the rewritten hash function. If there is no local file system, the default temporary file system is selected to save the file.

[0028] Furthermore, before the multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system to be fused, the following steps are also included:

[0029] If the fusion identification environment variable of the computing node to which the temporary file system belongs is a default value, marking the temporary file system as an independent temporary file system;

[0030] The process reads the corresponding storage node list file and file system priority configuration file based on the workflow name of the temporary file system to be merged, performs view fusion, and provides a unified metadata view. The process also includes:

[0031] If a file is written into the independent temporary file system, determining whether the file exists in the independent temporary file system;

[0032] If the file already exists in the independent temporary file system, directly save the file to the independent temporary file system;

[0033] If the file does not exist in the independent temporary file system, it is determined whether the computing node of the independent temporary file system has a local file system. If there is a local file system, the local file system is first selected to save the file, and then the hash value of the file path is calculated and the corresponding temporary file system is selected to save the file through the rewritten hash function. If there is no local file system, the default temporary file system is selected to save the file.

[0034] The present invention also proposes a multi-domain metadata management system for high-performance computing scenarios, including a multi-domain metadata register and multiple temporary file systems. The multi-domain metadata register is used to:

[0035] Obtain and save the workflow name, storage node list file and file system priority configuration file of each temporary file system;

[0036] Wait for and obtain the view fusion request of the temporary file system that needs to be merged, and read the corresponding storage node list file and file system priority configuration file according to the workflow name of the temporary file system that needs to be merged for view fusion to provide a unified metadata view.

[0037] Furthermore, the temporary file system includes a temporary file system server and a temporary file system client. The temporary file system server is used to retrieve the configuration information of the temporary file system and send it to the temporary file system client; the temporary file system client is used to obtain the configuration file of the multi-domain metadata register and parse it to obtain the address of the multi-domain metadata register, and then obtain the specified communication protocol variable in the network; and is also used to send a request for the specified communication protocol to the multi-domain metadata register.

[0038] Furthermore, the temporary file system client is further configured to read a fusion identification environment variable of a computing node to which the temporary file system belongs, and if the fusion identification environment variable is a specified value, send a view fusion request to the multi-domain metadata registrar.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] The present invention uses a multi-domain metadata registrar to store the configuration information of each temporary file system instance, and automatically provides a unified metadata view based on the configuration information of each temporary file system instance when needed, thereby only associating the independent temporary file system instances that need to be accessed and providing a unified metadata view, rather than taking a global synchronization operation for metadata changes of all file system instances. This reduces metadata synchronization overhead, optimizes metadata management, and improves file system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 2 is a system structure diagram of an embodiment of the present invention.

[0042] Figure 2 Flowchart of a method according to an embodiment of the present invention.

[0043] Figure 3 The figure is a schematic diagram of the process of writing a file in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0045] Example 1

[0046] In high-performance computing scenarios, existing distributed file systems need to ensure that different processes can obtain the latest file system namespace changes from other processes to maintain file system consistency. This requires frequent synchronization operations. Especially in large-scale distributed environments, these synchronization operations are very time-consuming, significantly affecting the file system's responsiveness and overall efficiency. In addition, global synchronization operations rely on network communication, and network latency and bandwidth limitations can also negatively impact synchronization efficiency. In large-scale high-performance computing applications, as the number of nodes and concurrent access volume continue to increase, network bottlenecks become increasingly prominent, further affecting the overall performance and stability of the file system.

[0047] The inventors found that in high-performance computing scenarios, operations mainly focus on temporary data, and the high availability of metadata and the strong consistency of the file system do not need to be considered. Therefore, a multi-domain metadata management method for high-performance computing scenarios is proposed, which is applied to a distributed file system including a multi-domain metadata registrar and multiple temporary file systems. The multi-domain metadata registrar is used to synchronously manage and maintain metadata, and metadata views are dynamically constructed in a differentiated manner. Related file system instances are managed according to needs, rather than performing global synchronization operations on all file instances. This decouples the consistency constraints between each other and effectively reduces system overhead.

[0048] like Figure 1 As shown, the distributed file system in this embodiment includes two parts: a multi-domain metadata register and a temporary file system, wherein:

[0049] The multi-domain metadata registrar is used to collect and manage the system environment where the user's workflow is located. When executing the preload library, the workflow name in the environment variable and the configuration information of the temporary file system are sent to the multi-domain metadata registrar. In addition, based on the workflow configuration information set by the user, the multi-domain metadata registrar automatically associates the separate temporary file system and generates the relevant metadata fusion view, thereby realizing unified management and efficient operation of the temporary file system.

[0050] The temporary file system consists of two parts: the server and the client, running in user space.

[0051] The temporary file system server consists of a key-value store, an I / O persistence layer, and an RPC (remote procedure call)-based communication layer. Running the temporary file system service requires configuring the local file and metadata storage path, the file system pseudo-mount path, and the storage path for the storage node list file. The pseudo-mount path does not store files but is used by the temporary file system client to intercept and process I / O requests under it.

[0052] The temporary file system client consists of an interception interface, file mapping, and an RPC-based communication layer. It is implemented in user space as a dynamic library by intercepting Linux system calls and is provided to users through a preloaded library method. The interception interface is responsible for intercepting application calls to the temporary file system and determining whether these calls are within the temporary file system namespace. Only operations within the pseudo-mount path are processed by the client; operations outside this range are passed to the kernel for processing. File mapping is used to manage file descriptors for open files and directories. The RPC communication layer is primarily used to transmit data and instructions between the client and server. The client can remotely call server functions to perform file system operations such as reading, writing, and metadata management. The first time you use the temporary file client, you must first execute the ls command to trigger the client to forward the current file system configuration information to the multi-domain metadata registration server. This ensures that the metadata registration server can correctly record and manage the configuration information of the file system instance, achieving efficient metadata synchronization and management.

[0053] The distributed file system in this embodiment stores file metadata in a local metadata storage engine, ensuring efficient metadata management and fast access. Text data is divided into chunks of equal size and stored in the local file system, enabling distributed data storage. When a client accesses a file, it is mapped to multiple server nodes through a hashing process, improving data access parallelism and overall system performance.

[0054] like Figure 2 As shown, the method of this embodiment includes the following steps:

[0055] S1) starting a multi-domain metadata registrar and generating a configuration file for the multi-domain metadata registrar;

[0056] S2) The temporary file system client obtains the configuration file of the multi-domain metadata register and parses it to obtain the address of the multi-domain metadata register, and then obtains the specified communication protocol variable in the network;

[0057] S3) The temporary file system client retrieves configuration information of the temporary file system;

[0058] S4) The multi-domain metadata registrar obtains and saves the workflow name, storage node list file, and file system priority configuration file of each temporary file system;

[0059] S5) A view fusion request of the temporary file system to be fused is sent to the multi-domain metadata registrar. The multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system to be fused, and reads the corresponding storage node list file and the file system priority configuration file according to the workflow name of the temporary file system to be fused to perform view fusion, so as to provide a unified metadata view.

[0060] Through the above steps, the method of this embodiment uses a multi-domain metadata registrar to store system information of a temporary file system instance, dynamically constructs a metadata view, reduces the synchronization scope, reduces the global synchronization management overhead and indexing overhead of metadata, and effectively improves the concurrent access performance of metadata in high-concurrency scenarios.

[0061] Each step is described in detail below.

[0062] In step S1 of this embodiment, starting the multi-domain metadata registrar includes: running the multi-domain metadata registrar program, starting the multi-domain metadata registration service, and generating a multi-domain metadata registrar configuration file. The multi-domain metadata registrar configuration file records the running address of the multi-domain metadata registrar service process to ensure that the client of the temporary file system can communicate with the multi-domain metadata registrar.

[0063] In step S2 of this embodiment, the communication protocol variable is designated as an RPC communication protocol variable. Step S2 includes the following specific implementation steps:

[0064] S2.1) When the temporary file system client initializes the environment, it obtains the path of the multi-domain metadata registrar configuration file according to the set environment variables;

[0065] S2.2) Read the metadata register configuration file content and parse the running address of the metadata register service process

[0066] S2.3) Set the context RPC communication protocol variable through the protocol extraction function extract_protocol.

[0067] Step S3 of this embodiment includes the following specific implementation steps:

[0068] First, set the workflow information for all temporary file systems;

[0069] Then, the temporary file system server retrieves the configuration information of the temporary file system and sends it to the temporary file system client, including the following specific implementation steps:

[0070] S3.1) When the temporary file system is initialized, a request is sent to the temporary file system server through the forward_get_fs_config function to retrieve the temporary file system configuration information;

[0071] S3.2) After receiving the request, the temporary file system server sends configuration information such as the file system mount path, local file and metadata storage path to the client.

[0072] Finally, the temporary file system client sends a request of a specified communication protocol to the multi-domain metadata registrar, which is an RPC request in this embodiment.

[0073] It should be noted that after step S3.2, if the temporary file system client is used for the first time, the temporary file system client must also send the configuration information of the temporary file system to the multi-domain metadata registrar via a specified communication protocol (RPC communication protocol in this embodiment) to ensure that the multi-domain metadata registrar can correctly record and manage the configuration information of the corresponding temporary file system instance.

[0074] This embodiment completes the initialization of the temporary file system client environment through steps S2 and S3. After the temporary file system client environment initialization is completed, an RPC request is sent to the multi-domain metadata registrar. In step S4 of this embodiment, after receiving the RPC request, the multi-domain metadata registrar stores the workflow name, storage node list file, and file system priority configuration file of the temporary file system instance. Each storage node list file corresponds to a view, and the storage node list file records the running address of the file system service process. The client needs to read the storage node list file to determine which server processes are included in this file system instance so that the client can communicate correctly.

[0075] In step S5 of this embodiment, when a view fusion request of a temporary file system to be fused is sent to the multi-domain metadata registrar, the temporary file system to be fused is specifically a temporary file system whose fusion flag environment variable is on. The specific implementation process includes:

[0076] In the specified computing node, set the fusion flag environment variable to the specified value ON;

[0077] The temporary file system client calls the request registry function request_registry to read the fusion identification environment variable of the computing node to which the temporary file system belongs;

[0078] If the fusion flag environment variable is the specified value on, the temporary file system is the temporary system that needs to be fused, the workflow information of the temporary file system that needs to be fused is configured, and the forward request registry function forward_request_registry is jumped to send a view fusion request to the multi-domain metadata register to fuse the workflow;

[0079] If the fusion identification environment variable of the computing node to which the temporary file system belongs is OFF, which is a default value, the temporary file system is marked as an independent temporary file system.

[0080] In step S5 of this embodiment, when performing view fusion based on the workflow name of the temporary file system to be fused, the corresponding storage node list file and the file system priority configuration file are read, which includes:

[0081] The multi-domain metadata registrar cyclically reads the storage node list files and file system priority configuration files corresponding to all workflows requesting fusion according to the workflow name of the temporary file system that needs to be merged, and splices them into a new storage node list file and file system priority configuration file. Then, the new storage node list file and file system priority configuration file are sent to the temporary file system that needs to be merged. After the client of the temporary file system that needs to be merged reads the new storage node list file and file system priority configuration file, it can operate on all workflows requesting fusion, thereby starting the view fusion function.

[0082] After step S5, this embodiment also includes a step of processing the written file in the metadata view, such as Figure 3 Shown, including:

[0083] S101) If a file is written into the metadata view, the process jumps to step S102; if a file is written into the independent temporary file system, the process jumps to step S103;

[0084] S102) determining whether the file exists in the temporary file system corresponding to the metadata view;

[0085] If the file already exists and only exists in a temporary file system, directly save the file to the temporary file system, end and exit;

[0086] If the file exists in multiple temporary file systems, select a corresponding temporary file system to save the file according to the file system priority in the temporary file system configuration information, and then exit;

[0087] If the file does not exist in any temporary file system, determine whether the computing node of the metadata view has a local file system. If so, first select the local file system to save the file. Then, calculate the hash value of the file path and select the corresponding temporary file system from the temporary file system corresponding to the metadata view using the rewritten hash function to save the file. The process ends and exits. If there is no local file system, select the default temporary file system in the temporary file system corresponding to the metadata view to save the file (in this embodiment, the first temporary file system in the configuration file of the metadata view is selected by default). The process ends and exits.

[0088] S103) determining whether the file exists in the independent temporary file system;

[0089] If the file already exists in the independent temporary file system, directly save the file to the independent temporary file system, end and exit;

[0090] If the file does not exist in the independent temporary file system, determine whether the computing node of the independent temporary file system has a local file system. If there is a local file system, first select the local file system to save the file, then calculate the hash value of the file path and select the corresponding temporary file system to save the file through the rewritten hash function. End and exit. If there is no local file system, select the default temporary file system to save the file. End and exit.

[0091] The following steps are used to verify the technical effect of this embodiment:

[0092] Step 1: Import the library file path, set the registrar configuration file storage path, and start the multi-domain metadata registrar service;

[0093] Step 2: After setting the local file storage path, metadata storage path, file system pseudo mount path, and storage node list file storage path, run the temporary file system service program and start multiple temporary file system instances;

[0094] Step 3: Set workflow information for all temporary file systems. When view fusion is not required, each temporary file system has corresponding workflow information as an independent temporary file system. Then, load the temporary file system client through the LD_PRELOAD environment variable, execute the ls command, forward the configuration information of the temporary file system to the multi-domain metadata registrar, and send the workflow name, storage node list file, and file system priority configuration file of all temporary file systems to the multi-domain metadata registrar for storage.

[0095] Step 4, write files to each temporary file system;

[0096] Step 5: Import environment variables on the new compute node, set the fusion flag environment variable to "on", configure the temporary file system workflow information to be fused, and enable the view fusion function.

[0097] Step 6: After executing the ls command using the LD_PRELOAD environment variable, the client of the temporary file system to be merged sends a view fusion request to the multi-domain metadata registrar. The multi-domain metadata registrar reads the storage node list file and file system configuration file of the temporary file system to be merged according to the workflow information configured in step 5, generates the storage node list file and file system configuration file for the merged view, and provides a unified metadata view. The file lists of all sub-views can be viewed on the terminal.

[0098] Step 7: Write the file to the fusion view. If the file already exists in multiple temporary file systems, select the temporary file system with the highest priority from all temporary file systems corresponding to the fusion view based on the priority information in the imported file system priority configuration file. Otherwise, first select the local file system of the corresponding computing node. After calculating the hash value of the file path, use the rewritten hash function to select the corresponding temporary file system from all temporary file systems in the fusion view. If no local file system is found, select the first default temporary file system from all temporary file systems in the fusion view according to the configuration file of the fusion view.

[0099] It can be seen that the above steps can effectively support multi-domain metadata management and transparent integration of metadata views, significantly improving metadata access performance and the overall efficiency of the file system.

[0100] Example 2

[0101] This embodiment provides a multi-domain metadata management system for high-performance computing scenarios corresponding to the method in the first embodiment, including a multi-domain metadata register and multiple temporary file systems. The multi-domain metadata register is used to:

[0102] Obtain and save the workflow name, storage node list file and file system priority configuration file of each temporary file system;

[0103] Wait for and obtain the view fusion request of the temporary file system that needs to be merged, and read the corresponding storage node list file and file system priority configuration file according to the workflow name of the temporary file system that needs to be merged for view fusion to provide a unified metadata view.

[0104] Similar to the temporary file system in the first embodiment, the temporary file system in this embodiment also includes a temporary file system server and a temporary file system client, wherein:

[0105] The temporary file system server is used to retrieve the configuration information of the temporary file system and send it to the temporary file system client;

[0106] The temporary file system client is used to obtain the configuration file of the multi-domain metadata registrar and parse it to obtain the address of the multi-domain metadata registrar, and then obtain the specified communication protocol variables in the network; it is also used to send a request for the specified communication protocol to the multi-domain metadata registrar; and it is also used to send the configuration information of the temporary file system to the multi-domain metadata registrar via the specified communication protocol when it is used for the first time.

[0107] Furthermore, in this embodiment, the temporary file system client is further configured to read a fusion identification environment variable of the computing node to which the temporary file system belongs, and if the fusion identification environment variable is a specified value, send a view fusion request to the multi-domain metadata registrar.

[0108] It should be understood by those skilled in the art that the embodiments of the present application may provide methods, systems, or computer program products. Therefore, the present application may take the form of an embodiment that combines software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code. The present application is based on the instructions generated by the flowcharts of the methods, devices (systems), and computer program products according to the embodiments of the present application for implementing the process. Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way.

[0109] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-domain metadata management method for high-performance computing scenarios, characterized by: Applied to a distributed file system, the distributed file system including a multi-domain metadata register and multiple temporary file systems, the method includes: The multi-domain metadata registrar obtains and saves the workflow name, storage node list file and file system priority configuration file of each temporary file system; The multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system that needs to be merged, and reads the corresponding storage node list file and file system priority configuration file according to the workflow name of the temporary file system that needs to be merged to perform view fusion to provide a unified metadata view.

2. The multi-domain metadata management method for high-performance computing scenarios according to claim 1, characterized in that: The temporary file system includes a temporary file system server and a temporary file system client. Before the multi-domain metadata register obtains and saves the workflow name, storage node list file and file system priority configuration file of each temporary file system, it includes: The temporary file system client obtains the configuration file of the multi-domain metadata register and parses it to obtain the address of the multi-domain metadata register, and then obtains the specified communication protocol variable in the network; The temporary file system server retrieves the configuration information of the temporary file system and sends it to the temporary file system client; The temporary file system client sends a request specifying a communication protocol to the multi-domain metadata registrar.

3. The multi-domain metadata management method for high-performance computing scenarios according to claim 2, characterized in that: After the temporary file system server retrieves the configuration information of the temporary file system and sends it to the temporary file system client, the method further includes: if the temporary file system client is used for the first time, the temporary file system client sends the configuration information of the temporary file system to the multi-domain metadata registrar through a specified communication protocol.

4. The multi-domain metadata management method for high-performance computing scenarios according to claim 2, characterized in that: Before the multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system that needs to be fused, it includes: In the specified compute node, set the fusion identity environment variable to the specified value; The temporary file system client reads the fusion identification environment variable of the computing node to which the temporary file system belongs, and sends a view fusion request to the multi-domain metadata registrar if the fusion identification environment variable is a specified value.

5. The multi-domain metadata management method for high-performance computing scenarios according to claim 1, characterized in that: When performing view fusion based on the workflow name of the temporary file system to be merged, the corresponding storage node list file and file system priority configuration file are read, including: According to the workflow name of the temporary file system that needs to be merged, the corresponding storage node list file and file system priority configuration file content are read cyclically, and spliced ​​into a new storage node list file and file system priority configuration file, and then the new storage node list file and file system priority configuration file are sent to the temporary file system that needs to be merged.

6. The multi-domain metadata management method for high-performance computing scenarios according to claim 1, characterized in that: The process reads the corresponding storage node list file and file system priority configuration file based on the workflow name of the temporary file system to be merged, performs view fusion, and provides a unified metadata view. The process also includes: If a file is written into the metadata view, determining whether the file exists in the temporary file system corresponding to the metadata view; If the file already exists and only exists in a temporary file system, directly save the file to the temporary file system; If the file exists in multiple temporary file systems, select a corresponding temporary file system to save the file according to the file system priority in the temporary file system configuration information; If the file does not exist in any temporary file system, it is determined whether the computing node of the metadata view has a local file system. If there is a local file system, the local file system is first selected to save the file, and then the hash value of the file path is calculated and the corresponding temporary file system is selected to save the file through the rewritten hash function. If there is no local file system, the default temporary file system is selected to save the file.

7. The multi-domain metadata management method for high-performance computing scenarios according to claim 1, characterized in that: Before the multi-domain metadata registrar waits for and obtains the view fusion request of the temporary file system that needs to be fused, it also includes: If the fusion identification environment variable of the computing node to which the temporary file system belongs is a default value, marking the temporary file system as an independent temporary file system; The process reads the corresponding storage node list file and file system priority configuration file based on the workflow name of the temporary file system to be merged, performs view fusion, and provides a unified metadata view. The process also includes: If a file is written into the independent temporary file system, determining whether the file exists in the independent temporary file system; If the file already exists in the independent temporary file system, directly save the file to the independent temporary file system; If the file does not exist in the independent temporary file system, it is determined whether the computing node of the independent temporary file system has a local file system. If there is a local file system, the local file system is first selected to save the file, and then the hash value of the file path is calculated and the corresponding temporary file system is selected to save the file through the rewritten hash function. If there is no local file system, the default temporary file system is selected to save the file.

8. A multi-domain metadata management system for high-performance computing scenarios, characterized by: The system comprises a multi-domain metadata register and a plurality of temporary file systems, wherein the multi-domain metadata register is used to: Obtain and save the workflow name, storage node list file and file system priority configuration file of each temporary file system; Wait for and obtain the view fusion request of the temporary file system that needs to be merged, and read the corresponding storage node list file and file system priority configuration file according to the workflow name of the temporary file system that needs to be merged for view fusion to provide a unified metadata view.

9. The multi-domain metadata management system for high-performance computing scenarios according to claim 8, characterized in that: The temporary file system includes a temporary file system server and a temporary file system client. The temporary file system server is used to retrieve the configuration information of the temporary file system and send it to the temporary file system client; the temporary file system client is used to obtain the configuration file of the multi-domain metadata register and parse it to obtain the address of the multi-domain metadata register, and then obtain the specified communication protocol variables in the network; and is also used to send a request for the specified communication protocol to the multi-domain metadata register.

10. The multi-domain metadata management system for high-performance computing scenarios according to claim 9, characterized in that: The temporary file system client is further configured to read a fusion identification environment variable of a computing node to which the temporary file system belongs, and if the fusion identification environment variable is a specified value, send a view fusion request to the multi-domain metadata register.

Images