Method, apparatus and device for virtual consistent multi-node message passing interface extension

Abstract

The application relates to a multi-node message passing interface extension method, device and equipment for virtual consistency. The method comprises the following steps: deploying a distributed file system client on each computing node of a multi-node cluster, creating a unified virtual directory mounting point, transparently converting a file operation request under the unified virtual directory mounting point into a corresponding network protocol, and connecting a global unified data storage pool, the unified virtual directory mounting point being used for providing a consistent file view for all computing nodes; prefixing a file read-write path of a message passing interface process to the unified virtual directory mounting point; in response to receiving a user job request, allocating a computing node for the user job request and ensuring that all allocated computing nodes have been mounted to the distributed file system, and starting the message passing interface process. The method can solve the consistency of cross-node file views from the system level without changing the communication logic.

Description

Technical Field

[0001] This application relates to the interdisciplinary fields of high-performance computing and geographic information science, and in particular to a method, apparatus, and device for extending a virtual consistency multi-node message passing interface. Background Technology

[0002] With the development of the intersection of high-performance computing and geographic information science, the demand for computing power for tasks such as large-scale terrain analysis and hydrological simulation is increasing. Parallel computing technology, with its distributed processing capabilities, has become a core support, which can break down complex tasks into multiple nodes for collaborative completion, and is suitable for high-resolution geographic data processing scenarios.

[0003] In traditional approaches, the scaling of parallel programs often relies on building simple clustered deployment schemes using network file systems.

[0004] However, traditional expansion methods have obvious technical drawbacks: they force all data to be transmitted through the central node, which means that data must be transmitted multiple times through the network, resulting in a loss of data locality; they also have problems such as fragmented resource scheduling and bandwidth contention. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for extending a virtual consistent multi-node message passing interface that can solve the consistency of cross-node file views at the system level without changing the communication logic, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for extending a virtual consistency multi-node message passing interface, applied to a scheduling management system, the scheduling management system including a resource abstraction layer, a computation execution layer, and a scheduling management layer, the method comprising:

[0007] The resource abstraction layer deploys a distributed file system client on each computing node of the multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0008] By configuring environment variables and / or configuration files and / or command-line parameters through the computation execution layer, the file read / write path prefix of the message passing interface process is pointed to the unified virtual directory mount point;

[0009] In response to a received user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process.

[0010] The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0011] In some embodiments of the method, the messaging interface process includes a terrain analysis messaging interface process based on a digital elevation model. The terrain analysis messaging interface process based on a digital elevation model is configured as a main process that reads the digital elevation model file from the input directory of the global unified data storage pool through the resource abstraction layer, then uses a region decomposition algorithm to divide the digital elevation model file into multiple continuous sub-region data blocks, and distributes each sub-region data block to the sub-process of the corresponding computing node through the scatter function of the messaging interface.

[0012] The terrain analysis message passing interface process based on the digital elevation model is also configured such that each subprocess performs at least one hydrological analysis calculation based on the sub-region data block, including depression filling, flow direction calculation, runoff accumulation calculation, and river network extraction. During the calculation process, the subprocess exchanges boundary raster data with the subprocesses on adjacent computing nodes through the sending and / or receiving functions of the message passing interface.

[0013] The terrain analysis message passing interface process based on the digital elevation model is also configured such that each subprocess writes the intermediate result file to the global unified data storage pool through the resource abstraction layer; after each subprocess completes its calculation, it sends its calculation result to the main process, or writes the final result file to the output directory of the global unified data storage pool in parallel.

[0014] In some embodiments of the method, the method further includes:

[0015] When multiple computing nodes concurrently write to different regions of the same intermediate or final result file, the distributed file system coordinates the write operations of each concurrent process through a distributed lock manager and / or byte range locks, determines the write data offset range of each concurrent process, and ensures the data consistency of the intermediate or final result file.

[0016] In some embodiments of the method, the step of allocating compute nodes for a user job request and ensuring that all allocated compute nodes are mounted to the distributed file system in response to receiving a user job request by the scheduling management layer, and starting a message passing interface process, includes:

[0017] Before starting the messaging interface process, ensure that all allocated compute nodes are mounted to the distributed file system and that the messaging interface task is set to a working directory or file path reference pointing to the unified virtual directory mount point.

[0018] In some embodiments of the method, the resource abstraction layer is further configured with a cache consistency mechanism, and the distributed file system client caches file data blocks and metadata in the memory of the computing node;

[0019] If a process on any computing node modifies the file content, the distributed file system server proactively notifies the distributed file system clients on other computing nodes that their caches have become invalid, so that all processes on the computing nodes can read the latest file data.

[0020] In some embodiments of the method, the intermediate result file includes at least one of a temporary flow direction matrix file generated during the iterative calculation process and an intermediate confluence data file, and the final result file includes at least one of a depression-free digital elevation model file, a river network extraction result file, and a watershed boundary file.

[0021] According to a second aspect of the present disclosure, a virtual consistency multi-node messaging interface extension apparatus is provided. The apparatus includes:

[0022] The resource abstraction module is used to deploy distributed file system clients on each computing node of a multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to a global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0023] The computation execution module is used to configure environment variables and / or configuration files and / or command line parameters, and to point the file read / write path prefix of the message passing interface process to the unified virtual directory mount point;

[0024] The scheduling and management module is used to respond to a received user job request, allocate computing nodes to the user job request, ensure that all allocated computing nodes are mounted to the distributed file system, and start the message passing interface process.

[0025] The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0026] According to a third aspect of the present disclosure, a computer device is provided. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0027] The resource abstraction layer deploys a distributed file system client on each computing node of the multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0028] By configuring environment variables and / or configuration files and / or command-line parameters through the computation execution layer, the file read / write path prefix of the message passing interface process is pointed to the unified virtual directory mount point;

[0029] In response to a received user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process.

[0030] The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0031] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0032] The resource abstraction layer deploys a distributed file system client on each computing node of the multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0033] By configuring environment variables and / or configuration files and / or command-line parameters through the computation execution layer, the file read / write path prefix of the message passing interface process is pointed to the unified virtual directory mount point;

[0034] In response to a received user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process.

[0035] The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0036] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0037] The resource abstraction layer deploys a distributed file system client on each computing node of the multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0038] By configuring environment variables and / or configuration files and / or command-line parameters through the computation execution layer, the file read / write path prefix of the message passing interface process is pointed to the unified virtual directory mount point;

[0039] In response to a received user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process.

[0040] The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0041] The virtual consistency multi-node message passing interface extension scheme provided in this application can solve the problem of inconsistent file views across nodes in a multi-node cluster from the system level. It does not require modification of the communication and calculation logic of the original message passing interface parallel program. The program can be transparently extended to the multi-node cluster through system configuration alone, which greatly reduces the development cost and complexity of program extension. At the same time, it avoids the drawback of data needing to be transmitted multiple times through the central node in the traditional solution, improves data transmission efficiency and the parallel processing capability of computing tasks, and is suitable for scenarios with high requirements for computing power and storage consistency, such as large-scale geographic data processing.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0044] Figure 1 This is a flowchart illustrating a virtual consistency multi-node messaging interface extension method according to an exemplary embodiment.

[0045] Figure 2 This is a schematic diagram illustrating the specific process of a virtual consistency multi-node messaging interface extension method according to an exemplary embodiment.

[0046] Figure 3 This is a structural block diagram of a virtual consistency multi-node messaging interface extension device according to an exemplary embodiment;

[0047] Figure 4 This is a diagram illustrating the internal structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In some implementations provided in this disclosure, the execution of the virtual consistency multi-node messaging interface extension method can be controlled by a unified controller or by multiple controllers. These controllers may include controllers on local terminals, such as the virtual consistency multi-node messaging interface extension controller, or controllers on remote servers, such as controllers in servers that can communicate with the virtual consistency multi-node messaging interface extension. In some implementations, the controllers on the local terminals and the controllers on the servers may jointly assist in completing the control processing of the virtual consistency multi-node messaging interface extension. The local terminals mentioned in this disclosure may include, but are not limited to, various robotic devices, in-vehicle devices, personal computers, laptops, smartphones, tablets, wearable devices, medical devices, VR (Virtual Reality) devices, etc. The servers may also be servers, server clusters, distributed subsystems, cloud processing platforms, servers containing blockchain nodes, and combinations thereof. The controllers described in this disclosure may include various control units capable of implementing logic processing functions, including but not limited to CPU (Central Processing Unit), PLC (Programmable Logic Controller), ECU (Electronic Control Unit), MCU (Microcontroller Unit), FPGA (Field Programmable Gate Array), and CPLD (Complex Programmable Logic Device), as well as controllers composed of one or more logic function units, chips, etc.

[0050] In some embodiments of this disclosure, a method for extending a virtual consistency multi-node message passing interface is provided, applied to a scheduling management system. The scheduling management system includes a resource abstraction layer, a computation execution layer, and a scheduling management layer. Figure 1 As shown, it includes the following steps:

[0051] S20. Deploy a distributed file system client on each computing node of the multi-node cluster through the resource abstraction layer. Create a unified virtual directory mount point in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes.

[0052] The resource abstraction layer is a functional layer deployed on each compute node of a multi-node cluster. It can create unified virtual directory mount points in the distributed file system client, translate file operation requests under this mount point into corresponding network protocols, and connect to a globally unified data storage pool, providing a consistent file access view for all compute nodes. The globally unified data storage pool is a collection of storage resources provided by the distributed storage cluster, used to store input data, intermediate result files, and final result files, supporting concurrent read and write operations by multiple node processes. A multi-node cluster refers to a computing cluster composed of multiple interconnected independent computers that work collaboratively to undertake parallel computing tasks. A compute node typically refers to an independent computer in a multi-node cluster that performs specific computing tasks and is the core computing unit of the cluster. Each compute node is equipped with hardware resources such as processors and memory, deploys a distributed file system client and related computing programs, receives data blocks distributed by the main process, executes computing tasks in parallel, and exchanges data and synchronizes results with other nodes through the network.

[0053] A distributed file system (DFS) typically refers to a software system that shares files and storage resources across multiple computer nodes, with data and metadata distributed across multiple physical locations within the cluster. It presents a unified file system interface to clients, supports concurrent access from multiple nodes, and uses specific network protocols to transmit and manage file data. A unified virtual directory mount point typically refers to a virtual directory path created by the distributed file system client on each compute node. All compute nodes share the same mount point path, and file operation requests under this path are automatically converted to the corresponding network protocol, connecting to a globally unified data storage pool. This ensures that processes on different nodes can see the same file directory structure and content. File operation requests typically refer to various operation instructions executed by processes on compute nodes on files, including reading file data, writing data to files, creating files, deleting files, and modifying file attributes. Transparent conversion typically refers to automatically converting file operation requests under the unified virtual directory mount point into network transmission protocols adapted to the globally unified data storage pool, without requiring additional process intervention. The conversion process is completely transparent to the processes; the processes do not need to know the actual storage location or transmission details of the data. A file view typically refers to the file system directory structure and file content status as seen by processes on compute nodes.

[0054] S22. Configure environment variables and / or configuration files and / or command line parameters through the computing execution layer to point the file read / write path prefix of the message passing interface process to the unified virtual directory mount point.

[0055] The computation execution layer is the functional layer that carries the message passing interface process. It includes the original parallel computing program distributed across multiple computing nodes. Through specific configuration, the file read and write paths of the processes are pointed to a unified virtual directory mount point, which can achieve multi-node expansion without modifying the internal logic of the program.

[0056] The file read / write path prefix usually refers to the beginning of the file read / write operation path, which is used to specify the base directory for file operations.

[0057] For message passing interface processes, by setting system environment variables, modifying program configuration files, or specifying command-line parameters during startup, the path prefix for all file read / write operations of the process can be uniformly pointed to the virtual directory mount point created by the resource abstraction layer. The internal logic of the original parallel computing program does not need to be modified. Its file operations, which originally pointed to local paths or other scattered paths, will automatically adapt to access to the unified mount point, thereby connecting to the global unified data storage pool and achieving cross-node consistency of file read / write.

[0058] S24. Upon receiving a user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process.

[0059] The scheduling management layer is responsible for receiving user job requests, allocating computing node resources, ensuring that the nodes are mounted with a distributed file system and the paths are correctly configured, and then starting the message passing interface process.

[0060] User job requests typically refer to specific computing task requests submitted by users to the cluster scheduling and management system.

[0061] After receiving a job request from a user, the job scheduling system server in the scheduling management layer can allocate suitable computing nodes to the job based on cluster resource utilization and job computation requirements. Before starting the message passing interface process, the scheduling system checks all allocated computing nodes to ensure that these nodes have successfully mounted the distributed file system and that the mounting status is normal.

[0062] S26. The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0063] Message passing interfaces are cross-language communication protocol standards used for communication and data exchange between different node processes in parallel computing programs.

[0064] In some embodiments of this disclosure, the problem of inconsistent file views across nodes in a multi-node cluster can be solved at the system level; the communication and computation logic of the original message passing interface parallel program can be modified without modifying the original message passing interface parallel program, and the program can be transparently extended to a multi-node cluster through system configuration alone, which greatly reduces the development cost and complexity of program extension; at the same time, it avoids the drawback of data needing to be transmitted multiple times through the central node in the traditional solution, improves data transmission efficiency and parallel processing capability of computing tasks, and is suitable for scenarios with high requirements for computing power and storage consistency, such as large-scale geographic data processing.

[0065] In some embodiments of this disclosure, the messaging interface process includes a terrain analysis messaging interface process based on a digital elevation model, as described above. Figure 2 The methods also include:

[0066] 202. The terrain analysis message passing interface process based on the digital elevation model is configured as the main process. After reading the digital elevation model file from the input directory of the global unified data storage pool through the resource abstraction layer, the digital elevation model file is divided into multiple continuous sub-region data blocks using a region decomposition algorithm. The sub-region data blocks are then distributed to the sub-processes of the corresponding computing nodes through the scatter function of the message passing interface.

[0067] 204. The terrain analysis message passing interface process based on the digital elevation model is further configured such that each subprocess performs at least one hydrological analysis calculation based on the sub-region data block, including depression filling, flow direction calculation, runoff accumulation calculation, and river network extraction. During the calculation process, the subprocess exchanges boundary raster data with the subprocesses on adjacent computing nodes through the sending and / or receiving functions of the message passing interface.

[0068] 206. The terrain analysis message passing interface process based on the digital elevation model is further configured such that each subprocess writes the intermediate result file to the global unified data storage pool through the resource abstraction layer; after each subprocess completes its calculation, it sends its calculation result to the main process, or writes the final result file to the output directory of the global unified data storage pool in parallel.

[0069] In some implementations, distributed file system client software can be installed and deployed on each compute node in a multi-node cluster. This software can be a parallel file system client adapted to high-performance computing scenarios or a distributed file system client compatible with the POSIX (Portable Operating System Interface) standard. After the client is installed, a unified virtual directory mount point is created in the operating system kernel or user space of each compute node. This mount point maintains the same path identifier across all compute nodes. When a process on any compute node performs read or write operations on files under this mount point, the client automatically converts these operation requests into the corresponding network transmission protocol and transmits them to the backend global unified data storage pool, achieving transparent processing of file operations and ensuring that processes on each node see a consistent file directory structure and content. For message passing interface processes, the path prefix for all file read and write operations of the process is uniformly pointed to the virtual directory mount point created by the resource abstraction layer through system environment variable settings, program configuration file modifications, or startup command line parameter specification. The internal logic of the original parallel computing program does not need to be modified. Its file operations, which originally pointed to local paths or other scattered paths, will automatically adapt to access to the unified mount point, thereby connecting to the global unified data storage pool and achieving cross-node consistency of file read and write operations.

[0070] After the message passing interface process starts, the main process accesses the global unified data storage pool through the resource abstraction layer to read the input data required for the job. The main process uses a suitable data analysis algorithm to divide the input data into multiple contiguous sub-region data blocks, and then distributes these data blocks to the child processes on various computing nodes through the message passing interface's distribution function. Each child process receives the data blocks and executes its corresponding computation task in parallel. During computation, if data interaction between adjacent regions is involved, the child processes exchange boundary data with child processes on adjacent computing nodes through the message passing interface's send and receive functions to ensure the accuracy of the computation results. Intermediate result files generated by the child processes during computation are written to the global unified data storage pool in real time through the resource abstraction layer. After the computation task is completed, each child process can send the final result to the main process for aggregation, or directly write the final result file to the specified output directory of the data storage pool in parallel.

[0071] In some examples, the TerrainAnalysis Using Digital Elevation Models Message Passing Interface (TauDEM MPI) is used as an example to illustrate its application in large-scale watershed distributed hydrological simulation scenarios. In this scenario, users need to perform hydrological analysis on a 1-meter resolution digital elevation model covering tens of thousands of square kilometers. Core computational tasks include depression filling, flow direction calculation, runoff accumulation calculation, and river network extraction. A single server's computing power cannot meet the timeliness requirements; therefore, an HPC (High-performance computing) cluster with dozens of computing nodes is used. The cluster deploys the Lustre distributed file system, an open-source parallel distributed file system designed for high-performance computing environments, capable of supporting large-scale data processing and high-concurrency access. A globally unified data storage pool is built. Each computing node creates a unified virtual directory mount point ` / mnt / shared_taudem` through a Lustre client deployed in the resource abstraction layer. All file operations on all nodes interface with the storage pool through this mount point. Users can upload the original digital elevation model (DEM) files and parameter files to the input directory of the global unified data storage pool (e.g., / mnt / shared_taudem / input) using file transfer tools. After receiving the user's job request, the scheduling management layer starts TauDEM MPI on the allocated compute node. The main process (Rank 0) reads the relevant files from the input directory through the distributed file system client of the resource abstraction layer.

[0072] The main process employs a region decomposition algorithm to divide the global digital elevation model (DEM) file and / or parameter file into multiple contiguous sub-region data blocks, ensuring the geographical continuity and balanced data volume of each block. Subsequently, the main process distributes each sub-region data block to its corresponding child processes on the computing nodes via the scatter function (MPI_Scatter) of the message passing interface. This distribution process is implemented through the cluster's internal network, transparent to the processes and without additional intervention. After receiving the sub-region data blocks, the child processes on each computing node perform hydrological analysis calculations in parallel, including depression filling, flow direction calculation, runoff accumulation calculation, and river network extraction. The calculation logic can be entirely reused from the existing TauDEM parallel program without any modifications. During the calculation process, due to the geographical adjacency of the sub-region data blocks, the child processes need to exchange boundary raster data with child processes on adjacent computing nodes to ensure the accuracy of the calculation results. At this point, the child process uses the send function (MPI_Send) and receive function (MPI_Recv) of the message passing interface to achieve bidirectional interaction of boundary raster data. The data exchange process is completed by relying on the cluster network and does not require additional configuration of the communication protocol.

[0073] In some implementations, the intermediate result file includes at least one of a temporary flow direction matrix file generated during the iterative calculation process and an intermediate confluence data file, and the final result file includes at least one of a depression-free digital elevation model file, a river network extraction result file, and a watershed boundary file.

[0074] During iterative computation, each subprocess generates intermediate result files such as temporary flow matrix files and intermediate confluence data files. These files are written in real-time to the temporary directory / mnt / shared_taudem / temp of the global unified data storage pool via the distributed file system client of the resource abstraction layer, ensuring the traceability and reusability of intermediate results and supporting breakpoint resumption. After all subprocesses complete all computation tasks, the final results are processed in parallel: each subprocess directly writes the final result data, including the depression-free digital elevation model file, river network extraction result file, and watershed boundary file for its assigned region, to the output directory / mnt / shared_taudem / output of the global unified data storage pool. Finally, these are aggregated to form a complete global result file, without requiring additional data integration by the main process.

[0075] In some embodiments of this disclosure, hydrological calculation tasks related to terrain analysis can be efficiently adapted to multi-node parallel environments; reasonable data partitioning is achieved through a region decomposition algorithm to ensure load balancing of each subprocess; efficient exchange of boundary data is achieved through the communication function of the message passing interface, ensuring the accuracy of hydrological analysis calculations; and flexible writing methods for intermediate and final results not only meet the needs of iterative calculations but also improve the storage and aggregation efficiency of result data, effectively supporting large-scale hydrological analysis tasks of high-resolution digital elevation models.

[0076] In some embodiments of this disclosure, the method further includes:

[0077] When multiple computing nodes concurrently write to different regions of the same intermediate or final result file, the distributed file system coordinates the write operations of each concurrent process through a distributed lock manager and / or byte range locks, determines the write data offset range of each concurrent process, and ensures the data consistency of the intermediate or final result file.

[0078] In some implementations, when processes on multiple computing nodes need to simultaneously write to different regions of the same intermediate or final result file, the distributed file system initiates a coordination mechanism. Through distributed lock management or byte range locking, the write operations of each process are scheduled in an orderly manner, clearly defining the offset range of the data written by each process within the file. This avoids conflicts between write operations from different processes, ensuring the integrity and consistency of the file data.

[0079] In some embodiments of this disclosure, the data conflict problem when multiple nodes concurrently write to the same file is resolved. Through the coordination mechanism of the distributed file system, the write scope of each process is clarified, ensuring the data consistency between intermediate result files and final result files. This avoids problems such as file data corruption and content disorder caused by concurrent writing, improves the reliability of parallel computing jobs, and enables processes of multiple nodes to safely and efficiently complete data writing operations in collaboration, providing a guarantee for file sharing writing in large-scale parallel computing.

[0080] In some embodiments of this disclosure, S24 includes:

[0081] Before starting the messaging interface process, ensure that all allocated compute nodes are mounted to the distributed file system and that the messaging interface task is set to a working directory or file path reference pointing to the unified virtual directory mount point.

[0082] In some implementations, after receiving a job request from a user, the job scheduling system server at the scheduling management layer can allocate suitable computing nodes to the job based on cluster resource usage and job computation requirements. Before starting the messaging interface process, the scheduling system checks all allocated computing nodes to ensure that these nodes have successfully mounted the distributed file system and that the mounting status is normal. Simultaneously, it confirms that the working directory or file path base of the messaging interface task correctly points to the unified virtual directory mount point to avoid job execution failure due to incorrect path configuration. After completing the above checks, the scheduling system starts the messaging interface process on each allocated node and begins executing the parallel computing job.

[0083] In some embodiments of this disclosure, by adding mount status checks and path configuration confirmation steps before process startup, the risk of job execution failure due to nodes not mounting file systems or incorrect path configuration is avoided in advance; thereby ensuring that all nodes participating in the computation are in a consistent operating environment, reducing abnormal interruptions during job execution, and thus improving the stability and success rate of parallel computing jobs.

[0084] In some embodiments of this disclosure, the resource abstraction layer is further configured with a cache consistency mechanism, and the distributed file system client caches file data blocks and metadata in the memory of the computing node;

[0085] If a process on any computing node modifies the file content, the distributed file system server proactively notifies the distributed file system clients on other computing nodes that their caches have become invalid, so that all processes on the computing nodes can read the latest file data.

[0086] In some implementations, the distributed file system client at the resource abstraction layer caches frequently accessed file data blocks and metadata in the memory of the compute nodes to improve file read efficiency. When a process on any compute node modifies the file content, the distributed file system server detects this change in real time and proactively sends a notification to all distributed file system clients on other compute nodes, informing them that the cache for the relevant file has expired. Upon receiving the notification, clients on other nodes will retrieve the latest file data from the globally unified data storage pool when reading the file subsequently, ensuring that all node processes can read consistent, up-to-date data.

[0087] In some embodiments of this disclosure, memory caching can be used to improve the response speed of file reading, while cache expiration notifications ensure that the file data read by all node processes is up-to-date; thereby avoiding computational logic errors caused by outdated cached data, balancing file access efficiency and data consistency, providing efficient and reliable support for frequent file reading operations during parallel computing, and further improving the performance of the entire parallel computing system.

[0088] The present disclosure provides several methods for extending the virtual consistency multi-node message passing interface, which can solve the problem of inconsistent file views across nodes in a multi-node cluster at the system level. It eliminates the need to modify the communication and computation logic of the original message passing interface parallel program, enabling transparent extension of the program to a multi-node cluster solely through system configuration, significantly reducing the development cost and complexity of program extension. Simultaneously, it avoids the drawback of traditional solutions requiring multiple data transmissions through a central node, improving data transmission efficiency and the parallel processing capability of computational tasks, and adapting to scenarios with high requirements for computing power and storage consistency, such as large-scale geographic data processing.

[0089] It is understood that the various embodiments of the methods described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Related details can be found in the descriptions of other method embodiments.

[0090] It should be understood that although the steps in the flowcharts shown in the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0091] Based on the description of the above-described embodiments of the virtual consistency multi-node message passing interface extension method, this disclosure also provides a virtual consistency multi-node message passing interface extension device for implementing the above-described virtual consistency multi-node message passing interface extension method. The device may include a system (including a distributed system), software (application), module, component, controller, server, terminal, etc., using the method described in the embodiments of this specification, combined with necessary implementation hardware. Based on the same innovative concept, the devices in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the devices are similar, the implementation of specific devices in the embodiments of this specification can refer to the implementation of the foregoing method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0092] Figure 3 This is a schematic block diagram illustrating a virtual consistency multi-node message passing interface extension device according to an exemplary embodiment. The device can be the aforementioned terminal, a server, or a module, component, device, control unit, etc., integrated into the terminal. For details, please refer to... Figure 3 The device 100 may include: a resource abstraction module 120, a computation execution module 140, and a scheduling management module 160. The resource abstraction module 120 is used to deploy a distributed file system client on each computing node of the multi-node cluster, and create a unified virtual directory mount point in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes. The computing execution module 140 is used to configure environment variables and / or configuration files and / or command line parameters, and to point the file read and write path prefix of the message passing interface process to the unified virtual directory mount point. The scheduling management module 160 is used to respond to the received user job request, allocate computing nodes for the user job request, ensure that all allocated computing nodes are mounted to the distributed file system, and start the message passing interface process. The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in chunks. The child processes perform parallel computing and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

[0093] Each module in the aforementioned virtual consistency multi-node message passing interface extension device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0094] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a virtual consistent multi-node message passing interface extension method.

[0095] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0096] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the virtual consistency multi-node message passing interface extension method described in any embodiment of this specification.

[0097] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by the processor of a computer device, enables the computer device to implement the virtual consistency multi-node message passing interface extension method as described in any embodiment of this disclosure.

[0098] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the virtual consistency multi-node message passing interface extension method described in any embodiment of this specification.

[0099] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0101] It should be noted that the apparatus, computer equipment, storage medium, and computer program products described above may also include other implementation methods according to the description of the method embodiments. Specific implementation methods can be found in the description of the relevant method embodiments. Furthermore, new embodiments formed by combinations of features from various methods, apparatuses, devices, and server embodiments still fall within the scope of this disclosure and will not be elaborated upon here.

[0102] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling and communication connections between the devices or units shown or described can be implemented through direct and / or indirect coupling / connection, through standard or custom interfaces or protocols, and can be implemented electrically, mechanically, or in other forms.

[0103] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0104] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for extending a virtual consistency multi-node message passing interface, characterized in that, Applied to a scheduling management system, which includes a resource abstraction layer, a computation execution layer, and a scheduling management layer, the method includes: The resource abstraction layer deploys a distributed file system client on each computing node of the multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to the global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes. By configuring environment variables and / or configuration files and / or command-line parameters through the computation execution layer, the file read / write path prefix of the message passing interface process is pointed to the unified virtual directory mount point; In response to a received user job request, the scheduling management layer allocates computing nodes to the user job request and ensures that all allocated computing nodes are mounted to the distributed file system, and starts the message passing interface process. The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

2. The method according to claim 1, characterized in that, The message passing interface process includes a terrain analysis message passing interface process based on a digital elevation model. The terrain analysis message passing interface process based on a digital elevation model is configured as a main process to read the digital elevation model file from the input directory of the global unified data storage pool through the resource abstraction layer, and then uses a region decomposition algorithm to divide the digital elevation model file into multiple continuous sub-region data blocks. The sub-region data blocks are then distributed to the sub-processes of the corresponding computing nodes through the scatter function of the message passing interface. The terrain analysis message passing interface process based on the digital elevation model is also configured such that each subprocess performs at least one hydrological analysis calculation based on the sub-region data block, including depression filling, flow direction calculation, runoff accumulation calculation, and river network extraction. During the calculation process, the subprocess exchanges boundary raster data with the subprocesses on adjacent computing nodes through the sending and / or receiving functions of the message passing interface. The terrain analysis message passing interface process based on the digital elevation model is also configured so that each subprocess writes the intermediate result file to the global unified data storage pool through the resource abstraction layer; After each subprocess completes its calculation, it sends its calculation result to the main process, or writes the final result file in parallel to the output directory of the global unified data storage pool.

3. The method according to claim 2, characterized in that, The method further includes: When multiple computing nodes concurrently write to different regions of the same intermediate or final result file, the distributed file system coordinates the write operations of each concurrent process through a distributed lock manager and / or byte range locks, determines the write data offset range of each concurrent process, and ensures the data consistency of the intermediate or final result file.

4. The method according to claim 1, characterized in that, The step of the scheduling management layer responding to a received user job request by allocating compute nodes for the user job request and ensuring that all allocated compute nodes are mounted to the distributed file system, and starting a message passing interface process, includes: Before starting the messaging interface process, ensure that all allocated compute nodes are mounted to the distributed file system and that the messaging interface task is set to a working directory or file path reference pointing to the unified virtual directory mount point.

5. The method according to claim 1, characterized in that, The resource abstraction layer is also configured with a cache consistency mechanism, and the distributed file system client caches file data blocks and metadata in the memory of the computing node; If a process on any computing node modifies the file content, the distributed file system server proactively notifies the distributed file system clients on other computing nodes that their caches have become invalid, so that all processes on the computing nodes can read the latest file data.

6. The method according to claim 2, characterized in that, The intermediate result file includes at least one of the temporary flow direction matrix file and intermediate confluence data file generated during the iterative calculation process, and the final result file includes at least one of the following: a depression-free digital elevation model file, a river network extraction result file, and a watershed boundary file.

7. A virtual consistency multi-node message passing interface extension device, characterized in that, The device includes: The resource abstraction module is used to deploy distributed file system clients on each computing node of a multi-node cluster. A unified virtual directory mount point is created in the distributed file system client. File operation requests under the unified virtual directory mount point are transparently converted into corresponding network protocols and connected to a global unified data storage pool. The unified virtual directory mount point is used to provide a consistent file view for all computing nodes. The computation execution module is used to configure environment variables and / or configuration files and / or command line parameters, and to point the file read / write path prefix of the message passing interface process to the unified virtual directory mount point; The scheduling and management module is used to respond to a received user job request, allocate computing nodes to the user job request, ensure that all allocated computing nodes are mounted to the distributed file system, and start the message passing interface process. The message passing interface process is configured as the main process to read input data from the global unified data storage pool through the resource abstraction layer and distribute it to each child process in segments. The child processes perform parallel computation and exchange boundary data through the message passing interface protocol. The intermediate result files and / or final result files of the child processes are written to the global unified data storage pool through the resource abstraction layer.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images